CIHealth Run Log

Environment: DEV

Time (UTC)	Job	Failed at	Result	PR	Failed tests	Details
23:56:45	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 8f8a842	n/a	passed	#5958 (open)	0	n/a
23:41:48	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 4e67bc7	alert, e2e	failed	★#6554 (open)	3	e2e (2) query-basic: command failed: exit status 1 23:41:48 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.001s. Expected success, but got an error: <errors.errorString \| 0xc000b66310>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[00:48:59.861]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:48:54.9335446Z);\nlet _endTime = datetime(2026-08-25T00:48:54.9335452Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-dh6t5tfmwl78'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:48:59.8208131Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;e378a004-3ace-4448-8d86-93b49cc20b45\",\r\n \"activityId\": \"3f29cef9-55f3-4e39-a44c-dc2745aeb329\",\r\n \"subActivityId\": \"9dc2e9b4-43ec-49f2-b454-4f12f219c5db\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"9b9edaeb-e277-470d-a49a-ba5e0f233409\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;e378a004-3ace-4448-8d86-93b49cc20b45 ARID=3f29cef9-55f3-4e39-a44c-dc2745aeb329 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/a5037b89-f047-4fed-bfb0-6c9466286aa0 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/a5037b89-f047-4fed-bfb0-6c9466286aa0 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/9b9edaeb-e277-470d-a49a-ba5e0f233409 \u003e DN.FE.ExecuteQuery/9dc2e9b4-43ec-49f2-b454-4f12f219c5db)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[00:48:59.984]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:48:54.9335446Z);\nlet _endTime = datetime(2026-08-25T00:48:54.9335452Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-dh6t5tfmwl78'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:48:59.8208131Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;e378a004-3ace-4448-8d86-93b49cc20b45\",\r\n \"activityId\": \"3f29cef9-55f3-4e39-a44c-dc2745aeb329\",\r\n \"subActivityId\": \"9dc2e9b4-43ec-49f2-b454-4f12f219c5db\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"9b9edaeb-e277-470d-a49a-ba5e0f233409\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;e378a004-3ace-4448-8d86-93b49cc20b45 ARID=3f29cef9-55f3-4e39-a44c-dc2745aeb329 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/a5037b89-f047-4fed-bfb0-6c9466286aa0 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/a5037b89-f047-4fed-bfb0-6c9466286aa0 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/9b9edaeb-e277-470d-a49a-ba5e0f233409 \u003e DN.FE.ExecuteQuery/9dc2e9b4-43ec-49f2-b454-4f12f219c5db)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[00:49:04.412]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:49:00.0668173Z);\nlet _endTime = datetime(2026-08-25T00:49:00.0668177Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-dh6t5tfmwl78'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:49:04.3919639Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;4c4f5480-a590-4f9a-a459-65dc9658f8ef\",\r\n \"activityId\": \"590d8d9f-c432-4b6c-a57a-14f6cfbdb0ea\",\r\n \"subActivityId\": \"f16f10a7-70e1-48c7-9e9d-209207cfc1e8\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"69739f9c-88ef-46ee-93e1-65af7b0a2416\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;4c4f5480-a590-4f9a-a459-65dc9658f8ef ARID=590d8d9f-c432-4b6c-a57a-14f6cfbdb0ea \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b66f5a6d-cdfd-418f-82f1-af8b0b7b5e03 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b66f5a6d-cdfd-418f-82f1-af8b0b7b5e03 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/69739f9c-88ef-46ee-93e1-65af7b0a2416 \u003e DN.FE.ExecuteQuery/f16f10a7-70e1-48c7-9e9d-209207cfc1e8)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[00:49:04.464]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:49:00.0668173Z);\nlet _endTime = datetime(2026-08-25T00:49:00.0668177Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-dh6t5tfmwl78'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:49:04.3919639Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;4c4f5480-a590-4f9a-a459-65dc9658f8ef\",\r\n \"activityId\": \"590d8d9f-c432-4b6c-a57a-14f6cfbdb0ea\",\r\n \"subActivityId\": \"f16f10a7-70e1-48c7-9e9d-209207cfc1e8\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"69739f9c-88ef-46ee-93e1-65af7b0a2416\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;4c4f5480-a590-4f9a-a459-65dc9658f8ef ARID=590d8d9f-c432-4b6c-a57a-14f6cfbdb0ea \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b66f5a6d-cdfd-418f-82f1-af8b0b7b5e03 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b66f5a6d-cdfd-418f-82f1-af8b0b7b5e03 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/69739f9c-88ef-46ee-93e1-65af7b0a2416 \u003e DN.FE.ExecuteQuery/f16f10a7-70e1-48c7-9e9d-209207cfc1e8)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[00:49:12.903]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:49:08.6625258Z);\nlet _endTime = datetime(2026-08-25T00:49:08.6625262Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-dh6t5tfmwl78'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:49:12.8815919Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a5c94555-e249-4511-a807-b8477933031f\",\r\n \"activityId\": \"3a1a3ba5-3fc9-4a7a-bbc2-c2b684f8ab2c\",\r\n \"subActivityId\": \"af4bd91d-24d7-4768-a750-045fb74a6811\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"833ff0b4-2415-4cdd-8b37-d9d4e7e62eaa\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a5c94555-e249-4511-a807-b8477933031f ARID=3a1a3ba5-3fc9-4a7a-bbc2-c2b684f8ab2c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/58e55353-9773-463e-ab07-afa8f479d884 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/58e55353-9773-463e-ab07-afa8f479d884 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/833ff0b4-2415-4cdd-8b37-d9d4e7e62eaa \u003e DN.FE.ExecuteQuery/af4bd91d-24d7-4768-a750-045fb74a6811)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[00:49:13.008]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:49:08.6625258Z);\nlet _endTime = datetime(2026-08-25T00:49:08.6625262Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-dh6t5tfmwl78'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:49:12.8815919Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a5c94555-e249-4511-a807-b8477933031f\",\r\n \"activityId\": \"3a1a3ba5-3fc9-4a7a-bbc2-c2b684f8ab2c\",\r\n \"subActivityId\": \"af4bd91d-24d7-4768-a750-045fb74a6811\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"833ff0b4-2415-4cdd-8b37-d9d4e7e62eaa\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a5c94555-e249-4511-a807-b8477933031f ARID=3a1a3ba5-3fc9-4a7a-bbc2-c2b684f8ab2c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/58e55353-9773-463e-ab07-afa8f479d884 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/58e55353-9773-463e-ab07-afa8f479d884 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/833ff0b4-2415-4cdd-8b37-d9d4e7e62eaa \u003e DN.FE.ExecuteQuery/af4bd91d-24d7-4768-a750-045fb74a6811)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.001s. Expected success, but got an error: <errors.errorString \| 0xc000b66310>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[00:48:59.861]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:48:54.9335446Z);\nlet _endTime = datetime(2026-08-25T00:48:54.9335452Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-dh6t5tfmwl78'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:48:59.8208131Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;e378a004-3ace-4448-8d86-93b49cc20b45\",\r\n \"activityId\": \"3f29cef9-55f3-4e39-a44c-dc2745aeb329\",\r\n \"subActivityId\": \"9dc2e9b4-43ec-49f2-b454-4f12f219c5db\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"9b9edaeb-e277-470d-a49a-ba5e0f233409\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;e378a004-3ace-4448-8d86-93b49cc20b45 ARID=3f29cef9-55f3-4e39-a44c-dc2745aeb329 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/a5037b89-f047-4fed-bfb0-6c9466286aa0 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/a5037b89-f047-4fed-bfb0-6c9466286aa0 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/9b9edaeb-e277-470d-a49a-ba5e0f233409 \u003e DN.FE.ExecuteQuery/9dc2e9b4-43ec-49f2-b454-4f12f219c5db)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[00:48:59.984]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:48:54.9335446Z);\nlet _endTime = datetime(2026-08-25T00:48:54.9335452Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-dh6t5tfmwl78'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:48:59.8208131Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;e378a004-3ace-4448-8d86-93b49cc20b45\",\r\n \"activityId\": \"3f29cef9-55f3-4e39-a44c-dc2745aeb329\",\r\n \"subActivityId\": \"9dc2e9b4-43ec-49f2-b454-4f12f219c5db\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"9b9edaeb-e277-470d-a49a-ba5e0f233409\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;e378a004-3ace-4448-8d86-93b49cc20b45 ARID=3f29cef9-55f3-4e39-a44c-dc2745aeb329 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/a5037b89-f047-4fed-bfb0-6c9466286aa0 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/a5037b89-f047-4fed-bfb0-6c9466286aa0 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/9b9edaeb-e277-470d-a49a-ba5e0f233409 \u003e DN.FE.ExecuteQuery/9dc2e9b4-43ec-49f2-b454-4f12f219c5db)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[00:49:04.412]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:49:00.0668173Z);\nlet _endTime = datetime(2026-08-25T00:49:00.0668177Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-dh6t5tfmwl78'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:49:04.3919639Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;4c4f5480-a590-4f9a-a459-65dc9658f8ef\",\r\n \"activityId\": \"590d8d9f-c432-4b6c-a57a-14f6cfbdb0ea\",\r\n \"subActivityId\": \"f16f10a7-70e1-48c7-9e9d-209207cfc1e8\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"69739f9c-88ef-46ee-93e1-65af7b0a2416\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;4c4f5480-a590-4f9a-a459-65dc9658f8ef ARID=590d8d9f-c432-4b6c-a57a-14f6cfbdb0ea \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b66f5a6d-cdfd-418f-82f1-af8b0b7b5e03 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b66f5a6d-cdfd-418f-82f1-af8b0b7b5e03 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/69739f9c-88ef-46ee-93e1-65af7b0a2416 \u003e DN.FE.ExecuteQuery/f16f10a7-70e1-48c7-9e9d-209207cfc1e8)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[00:49:04.464]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:49:00.0668173Z);\nlet _endTime = datetime(2026-08-25T00:49:00.0668177Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-dh6t5tfmwl78'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:49:04.3919639Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;4c4f5480-a590-4f9a-a459-65dc9658f8ef\",\r\n \"activityId\": \"590d8d9f-c432-4b6c-a57a-14f6cfbdb0ea\",\r\n \"subActivityId\": \"f16f10a7-70e1-48c7-9e9d-209207cfc1e8\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"69739f9c-88ef-46ee-93e1-65af7b0a2416\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;4c4f5480-a590-4f9a-a459-65dc9658f8ef ARID=590d8d9f-c432-4b6c-a57a-14f6cfbdb0ea \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b66f5a6d-cdfd-418f-82f1-af8b0b7b5e03 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b66f5a6d-cdfd-418f-82f1-af8b0b7b5e03 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/69739f9c-88ef-46ee-93e1-65af7b0a2416 \u003e DN.FE.ExecuteQuery/f16f10a7-70e1-48c7-9e9d-209207cfc1e8)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[00:49:12.903]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:49:08.6625258Z);\nlet _endTime = datetime(2026-08-25T00:49:08.6625262Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-dh6t5tfmwl78'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:49:12.8815919Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a5c94555-e249-4511-a807-b8477933031f\",\r\n \"activityId\": \"3a1a3ba5-3fc9-4a7a-bbc2-c2b684f8ab2c\",\r\n \"subActivityId\": \"af4bd91d-24d7-4768-a750-045fb74a6811\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"833ff0b4-2415-4cdd-8b37-d9d4e7e62eaa\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a5c94555-e249-4511-a807-b8477933031f ARID=3a1a3ba5-3fc9-4a7a-bbc2-c2b684f8ab2c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/58e55353-9773-463e-ab07-afa8f479d884 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/58e55353-9773-463e-ab07-afa8f479d884 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/833ff0b4-2415-4cdd-8b37-d9d4e7e62eaa \u003e DN.FE.ExecuteQuery/af4bd91d-24d7-4768-a750-045fb74a6811)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[00:49:13.008]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:49:08.6625258Z);\nlet _endTime = datetime(2026-08-25T00:49:08.6625262Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-dh6t5tfmwl78'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:49:12.8815919Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a5c94555-e249-4511-a807-b8477933031f\",\r\n \"activityId\": \"3a1a3ba5-3fc9-4a7a-bbc2-c2b684f8ab2c\",\r\n \"subActivityId\": \"af4bd91d-24d7-4768-a750-045fb74a6811\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"833ff0b4-2415-4cdd-8b37-d9d4e7e62eaa\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a5c94555-e249-4511-a807-b8477933031f ARID=3a1a3ba5-3fc9-4a7a-bbc2-c2b684f8ab2c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/58e55353-9773-463e-ab07-afa8f479d884 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/58e55353-9773-463e-ab07-afa8f479d884 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/833ff0b4-2415-4cdd-8b37-d9d4e7e62eaa \u003e DN.FE.ExecuteQuery/af4bd91d-24d7-4768-a750-045fb74a6811)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 23:41:48 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-h74snzg5tpbh/cluster-candidate-5-0-gp9ztx should provision Unexpected error: <fmt.wrapError \| 0xc0014a05c0>: failed to create HCP cluster cluster-candidate-5-0-gp9ztx: failed waiting for cluster="cluster-candidate-5-0-gp9ztx" in resourcegroup="rg-candidate-5-0-h74snzg5tpbh" to finish creating: GET https://rp.j1843200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/de4739f4-eebd-4fe2-bf80-3bf6d57ff617 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/de4739f4-eebd-4fe2-bf80-3bf6d57ff617", "name": "de4739f4-eebd-4fe2-bf80-3bf6d57ff617", "status": "Failed", "startTime": "2026-08-25T00:23:47.993778746Z", "endTime": "2026-08-25T00:42:49.076566651Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-h74snzg5tpbh/cluster-candidate-5-0-gp9ztx should provision Unexpected error: <fmt.wrapError \| 0xc0014a05c0>: failed to create HCP cluster cluster-candidate-5-0-gp9ztx: failed waiting for cluster="cluster-candidate-5-0-gp9ztx" in resourcegroup="rg-candidate-5-0-h74snzg5tpbh" to finish creating: GET https://rp.j1843200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/de4739f4-eebd-4fe2-bf80-3bf6d57ff617 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/de4739f4-eebd-4fe2-bf80-3bf6d57ff617", "name": "de4739f4-eebd-4fe2-bf80-3bf6d57ff617", "status": "Failed", "startTime": "2026-08-25T00:23:47.993778746Z", "endTime": "2026-08-25T00:42:49.076566651Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 23:41:48 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-25T00:35:25Z Ended: 2026-08-25T00:56:19Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1843200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd9551d19b6bblvufm3l3a421htavmi-a4k0z5e8x0e3t0f", pod="router-597d99bc98-tlmg9", severity="warning" Description: Pod ocm-arohcpci01-2sd9551d19b6bblvufm3l3a421htavmi-a4k0z5e8x0e3t0f/router-597d99bc98-tlmg9 has been in a non-ready state for longer than 5 minutes.
23:30:58	pull-ci-Azure-ARO-HCP-main-e2e-parallel head d5a9f48	alert, e2e	failed	★#6648 (open)	2	e2e (1) query-basic: command failed: exit status 1 23:30:58 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.009s. Expected success, but got an error: <errors.errorString \| 0xc00083c410>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[00:23:58.357]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:23:55.0351655Z);\nlet _endTime = datetime(2026-08-25T00:23:55.0351661Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-b5s7zllhqcpk'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:23:58.3330999Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;125f7c3f-8e3c-4adc-92d3-11f56e27a65b\",\r\n \"activityId\": \"442abb20-5490-45f5-ba70-3bca3bed0639\",\r\n \"subActivityId\": \"b9132e8a-1c55-4f09-8982-f344852f0eee\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"99a04993-dace-423f-a879-2fbeee2a7b84\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;125f7c3f-8e3c-4adc-92d3-11f56e27a65b ARID=442abb20-5490-45f5-ba70-3bca3bed0639 \u003e GW.Http.CallContext/442abb20-5490-45f5-ba70-3bca3bed0639 \u003e GW.ExecuteQuery/99a04993-dace-423f-a879-2fbeee2a7b84 \u003e DN.FE.ExecuteQuery/b9132e8a-1c55-4f09-8982-f344852f0eee)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[00:23:58.411]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:23:55.0351655Z);\nlet _endTime = datetime(2026-08-25T00:23:55.0351661Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-b5s7zllhqcpk'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:23:58.3330999Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;125f7c3f-8e3c-4adc-92d3-11f56e27a65b\",\r\n \"activityId\": \"442abb20-5490-45f5-ba70-3bca3bed0639\",\r\n \"subActivityId\": \"b9132e8a-1c55-4f09-8982-f344852f0eee\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"99a04993-dace-423f-a879-2fbeee2a7b84\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;125f7c3f-8e3c-4adc-92d3-11f56e27a65b ARID=442abb20-5490-45f5-ba70-3bca3bed0639 \u003e GW.Http.CallContext/442abb20-5490-45f5-ba70-3bca3bed0639 \u003e GW.ExecuteQuery/99a04993-dace-423f-a879-2fbeee2a7b84 \u003e DN.FE.ExecuteQuery/b9132e8a-1c55-4f09-8982-f344852f0eee)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[00:24:01.787]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:23:58.4821435Z);\nlet _endTime = datetime(2026-08-25T00:23:58.4821442Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-b5s7zllhqcpk'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:24:01.7618170Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;7425e14f-788f-421a-b19d-fdd4f5d966b5\",\r\n \"activityId\": \"d2adae1e-7609-4016-b166-5a8ff8be55d8\",\r\n \"subActivityId\": \"7c0e685c-e2d3-436f-89bc-9d9fa3141b61\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"8cf560a3-f257-4d9d-900b-aaf46185fe8b\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;7425e14f-788f-421a-b19d-fdd4f5d966b5 ARID=d2adae1e-7609-4016-b166-5a8ff8be55d8 \u003e GW.Http.CallContext/d2adae1e-7609-4016-b166-5a8ff8be55d8 \u003e GW.ExecuteQuery/8cf560a3-f257-4d9d-900b-aaf46185fe8b \u003e DN.FE.ExecuteQuery/7c0e685c-e2d3-436f-89bc-9d9fa3141b61)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[00:24:01.845]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:23:58.4821435Z);\nlet _endTime = datetime(2026-08-25T00:23:58.4821442Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-b5s7zllhqcpk'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:24:01.7618170Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;7425e14f-788f-421a-b19d-fdd4f5d966b5\",\r\n \"activityId\": \"d2adae1e-7609-4016-b166-5a8ff8be55d8\",\r\n \"subActivityId\": \"7c0e685c-e2d3-436f-89bc-9d9fa3141b61\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"8cf560a3-f257-4d9d-900b-aaf46185fe8b\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;7425e14f-788f-421a-b19d-fdd4f5d966b5 ARID=d2adae1e-7609-4016-b166-5a8ff8be55d8 \u003e GW.Http.CallContext/d2adae1e-7609-4016-b166-5a8ff8be55d8 \u003e GW.ExecuteQuery/8cf560a3-f257-4d9d-900b-aaf46185fe8b \u003e DN.FE.ExecuteQuery/7c0e685c-e2d3-436f-89bc-9d9fa3141b61)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[00:24:08.682]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:24:05.4122339Z);\nlet _endTime = datetime(2026-08-25T00:24:05.4122344Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-b5s7zllhqcpk'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:24:08.6621602Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;6e8eb1dd-b3b2-4c08-bb8a-fabaa900a66f\",\r\n \"activityId\": \"ed3e04f4-6a49-4e81-9b00-d4558d9cddcc\",\r\n \"subActivityId\": \"4cad9bba-6029-42b0-a77f-5c14bc4e8a84\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"58dfa59a-bdfb-430f-af23-104aaad94cc9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;6e8eb1dd-b3b2-4c08-bb8a-fabaa900a66f ARID=ed3e04f4-6a49-4e81-9b00-d4558d9cddcc \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/7d4daa74-783c-457a-8256-bc6951fb2784 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/7d4daa74-783c-457a-8256-bc6951fb2784 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/58dfa59a-bdfb-430f-af23-104aaad94cc9 \u003e DN.FE.ExecuteQuery/4cad9bba-6029-42b0-a77f-5c14bc4e8a84)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[00:24:08.778]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:24:05.4122339Z);\nlet _endTime = datetime(2026-08-25T00:24:05.4122344Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-b5s7zllhqcpk'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:24:08.6621602Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;6e8eb1dd-b3b2-4c08-bb8a-fabaa900a66f\",\r\n \"activityId\": \"ed3e04f4-6a49-4e81-9b00-d4558d9cddcc\",\r\n \"subActivityId\": \"4cad9bba-6029-42b0-a77f-5c14bc4e8a84\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"58dfa59a-bdfb-430f-af23-104aaad94cc9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;6e8eb1dd-b3b2-4c08-bb8a-fabaa900a66f ARID=ed3e04f4-6a49-4e81-9b00-d4558d9cddcc \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/7d4daa74-783c-457a-8256-bc6951fb2784 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/7d4daa74-783c-457a-8256-bc6951fb2784 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/58dfa59a-bdfb-430f-af23-104aaad94cc9 \u003e DN.FE.ExecuteQuery/4cad9bba-6029-42b0-a77f-5c14bc4e8a84)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.009s. Expected success, but got an error: <errors.errorString \| 0xc00083c410>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[00:23:58.357]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:23:55.0351655Z);\nlet _endTime = datetime(2026-08-25T00:23:55.0351661Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-b5s7zllhqcpk'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:23:58.3330999Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;125f7c3f-8e3c-4adc-92d3-11f56e27a65b\",\r\n \"activityId\": \"442abb20-5490-45f5-ba70-3bca3bed0639\",\r\n \"subActivityId\": \"b9132e8a-1c55-4f09-8982-f344852f0eee\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"99a04993-dace-423f-a879-2fbeee2a7b84\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;125f7c3f-8e3c-4adc-92d3-11f56e27a65b ARID=442abb20-5490-45f5-ba70-3bca3bed0639 \u003e GW.Http.CallContext/442abb20-5490-45f5-ba70-3bca3bed0639 \u003e GW.ExecuteQuery/99a04993-dace-423f-a879-2fbeee2a7b84 \u003e DN.FE.ExecuteQuery/b9132e8a-1c55-4f09-8982-f344852f0eee)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[00:23:58.411]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:23:55.0351655Z);\nlet _endTime = datetime(2026-08-25T00:23:55.0351661Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-b5s7zllhqcpk'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:23:58.3330999Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;125f7c3f-8e3c-4adc-92d3-11f56e27a65b\",\r\n \"activityId\": \"442abb20-5490-45f5-ba70-3bca3bed0639\",\r\n \"subActivityId\": \"b9132e8a-1c55-4f09-8982-f344852f0eee\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"99a04993-dace-423f-a879-2fbeee2a7b84\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;125f7c3f-8e3c-4adc-92d3-11f56e27a65b ARID=442abb20-5490-45f5-ba70-3bca3bed0639 \u003e GW.Http.CallContext/442abb20-5490-45f5-ba70-3bca3bed0639 \u003e GW.ExecuteQuery/99a04993-dace-423f-a879-2fbeee2a7b84 \u003e DN.FE.ExecuteQuery/b9132e8a-1c55-4f09-8982-f344852f0eee)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[00:24:01.787]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:23:58.4821435Z);\nlet _endTime = datetime(2026-08-25T00:23:58.4821442Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-b5s7zllhqcpk'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:24:01.7618170Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;7425e14f-788f-421a-b19d-fdd4f5d966b5\",\r\n \"activityId\": \"d2adae1e-7609-4016-b166-5a8ff8be55d8\",\r\n \"subActivityId\": \"7c0e685c-e2d3-436f-89bc-9d9fa3141b61\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"8cf560a3-f257-4d9d-900b-aaf46185fe8b\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;7425e14f-788f-421a-b19d-fdd4f5d966b5 ARID=d2adae1e-7609-4016-b166-5a8ff8be55d8 \u003e GW.Http.CallContext/d2adae1e-7609-4016-b166-5a8ff8be55d8 \u003e GW.ExecuteQuery/8cf560a3-f257-4d9d-900b-aaf46185fe8b \u003e DN.FE.ExecuteQuery/7c0e685c-e2d3-436f-89bc-9d9fa3141b61)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[00:24:01.845]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:23:58.4821435Z);\nlet _endTime = datetime(2026-08-25T00:23:58.4821442Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-b5s7zllhqcpk'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:24:01.7618170Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;7425e14f-788f-421a-b19d-fdd4f5d966b5\",\r\n \"activityId\": \"d2adae1e-7609-4016-b166-5a8ff8be55d8\",\r\n \"subActivityId\": \"7c0e685c-e2d3-436f-89bc-9d9fa3141b61\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"8cf560a3-f257-4d9d-900b-aaf46185fe8b\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;7425e14f-788f-421a-b19d-fdd4f5d966b5 ARID=d2adae1e-7609-4016-b166-5a8ff8be55d8 \u003e GW.Http.CallContext/d2adae1e-7609-4016-b166-5a8ff8be55d8 \u003e GW.ExecuteQuery/8cf560a3-f257-4d9d-900b-aaf46185fe8b \u003e DN.FE.ExecuteQuery/7c0e685c-e2d3-436f-89bc-9d9fa3141b61)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[00:24:08.682]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:24:05.4122339Z);\nlet _endTime = datetime(2026-08-25T00:24:05.4122344Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-b5s7zllhqcpk'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:24:08.6621602Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;6e8eb1dd-b3b2-4c08-bb8a-fabaa900a66f\",\r\n \"activityId\": \"ed3e04f4-6a49-4e81-9b00-d4558d9cddcc\",\r\n \"subActivityId\": \"4cad9bba-6029-42b0-a77f-5c14bc4e8a84\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"58dfa59a-bdfb-430f-af23-104aaad94cc9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;6e8eb1dd-b3b2-4c08-bb8a-fabaa900a66f ARID=ed3e04f4-6a49-4e81-9b00-d4558d9cddcc \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/7d4daa74-783c-457a-8256-bc6951fb2784 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/7d4daa74-783c-457a-8256-bc6951fb2784 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/58dfa59a-bdfb-430f-af23-104aaad94cc9 \u003e DN.FE.ExecuteQuery/4cad9bba-6029-42b0-a77f-5c14bc4e8a84)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[00:24:08.778]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:24:05.4122339Z);\nlet _endTime = datetime(2026-08-25T00:24:05.4122344Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-b5s7zllhqcpk'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:24:08.6621602Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;6e8eb1dd-b3b2-4c08-bb8a-fabaa900a66f\",\r\n \"activityId\": \"ed3e04f4-6a49-4e81-9b00-d4558d9cddcc\",\r\n \"subActivityId\": \"4cad9bba-6029-42b0-a77f-5c14bc4e8a84\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"58dfa59a-bdfb-430f-af23-104aaad94cc9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;6e8eb1dd-b3b2-4c08-bb8a-fabaa900a66f ARID=ed3e04f4-6a49-4e81-9b00-d4558d9cddcc \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/7d4daa74-783c-457a-8256-bc6951fb2784 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/7d4daa74-783c-457a-8256-bc6951fb2784 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/58dfa59a-bdfb-430f-af23-104aaad94cc9 \u003e DN.FE.ExecuteQuery/4cad9bba-6029-42b0-a77f-5c14bc4e8a84)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... alert (1) alert [svc] BackendControllerRetryHotLoop fired 23:30:58 UTC · failed at alert · test [aro-hcp-observability] [svc] alert BackendControllerRetryHotLoop does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-25T00:10:04Z Ended: 2026-08-25T00:34:04Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j8026752-svc", component="backend", name="observemanagedresourcegroup", severity="warning" Description: Backend controller workqueue observemanagedresourcegroup has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work.
23:25:09	pull-ci-Azure-ARO-HCP-main-e2e-parallel head a660b6e	e2e	failed	★#6660 (open)	1	e2e (1) query-basic: command failed: exit status 1 23:25:09 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.003s. Expected success, but got an error: <errors.errorString \| 0xc000626a40>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[00:14:56.798]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:14:53.4068554Z);\nlet _endTime = datetime(2026-08-25T00:14:53.4068559Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-jk7hknvqffmd'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:14:56.7779164Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;cb2874aa-23b9-4e90-82b8-8dbdc8ccb12d\",\r\n \"activityId\": \"697030e7-6169-48d4-afab-e55af5423283\",\r\n \"subActivityId\": \"7025f3d5-21ad-4b66-b1ec-6016c4d8ba2f\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"e5629797-dc74-431c-89f2-26cc2ad935e1\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;cb2874aa-23b9-4e90-82b8-8dbdc8ccb12d ARID=697030e7-6169-48d4-afab-e55af5423283 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/d94d70a1-2ca3-4462-b247-1c225640a5b0 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/d94d70a1-2ca3-4462-b247-1c225640a5b0 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/e5629797-dc74-431c-89f2-26cc2ad935e1 \u003e DN.FE.ExecuteQuery/7025f3d5-21ad-4b66-b1ec-6016c4d8ba2f)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[00:14:56.846]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:14:53.4068554Z);\nlet _endTime = datetime(2026-08-25T00:14:53.4068559Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-jk7hknvqffmd'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:14:56.7779164Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;cb2874aa-23b9-4e90-82b8-8dbdc8ccb12d\",\r\n \"activityId\": \"697030e7-6169-48d4-afab-e55af5423283\",\r\n \"subActivityId\": \"7025f3d5-21ad-4b66-b1ec-6016c4d8ba2f\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"e5629797-dc74-431c-89f2-26cc2ad935e1\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;cb2874aa-23b9-4e90-82b8-8dbdc8ccb12d ARID=697030e7-6169-48d4-afab-e55af5423283 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/d94d70a1-2ca3-4462-b247-1c225640a5b0 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/d94d70a1-2ca3-4462-b247-1c225640a5b0 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/e5629797-dc74-431c-89f2-26cc2ad935e1 \u003e DN.FE.ExecuteQuery/7025f3d5-21ad-4b66-b1ec-6016c4d8ba2f)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[00:15:00.265]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:14:56.9537642Z);\nlet _endTime = datetime(2026-08-25T00:14:56.9537648Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-jk7hknvqffmd'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:15:00.2458657Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;d3045808-57d8-4c23-8f73-affb30f6a130\",\r\n \"activityId\": \"60865211-a5af-4242-8bd8-14b7a1f199ba\",\r\n \"subActivityId\": \"865c926e-620f-4107-9fd8-43ac9ee494b3\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"70a56a06-7a14-4b6c-b645-990ed51e93ae\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;d3045808-57d8-4c23-8f73-affb30f6a130 ARID=60865211-a5af-4242-8bd8-14b7a1f199ba \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/230b95d6-3e5a-4f20-bb04-f8704206f9bc \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/230b95d6-3e5a-4f20-bb04-f8704206f9bc \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/70a56a06-7a14-4b6c-b645-990ed51e93ae \u003e DN.FE.ExecuteQuery/865c926e-620f-4107-9fd8-43ac9ee494b3)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[00:15:00.327]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:14:56.9537642Z);\nlet _endTime = datetime(2026-08-25T00:14:56.9537648Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-jk7hknvqffmd'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:15:00.2458657Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;d3045808-57d8-4c23-8f73-affb30f6a130\",\r\n \"activityId\": \"60865211-a5af-4242-8bd8-14b7a1f199ba\",\r\n \"subActivityId\": \"865c926e-620f-4107-9fd8-43ac9ee494b3\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"70a56a06-7a14-4b6c-b645-990ed51e93ae\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;d3045808-57d8-4c23-8f73-affb30f6a130 ARID=60865211-a5af-4242-8bd8-14b7a1f199ba \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/230b95d6-3e5a-4f20-bb04-f8704206f9bc \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/230b95d6-3e5a-4f20-bb04-f8704206f9bc \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/70a56a06-7a14-4b6c-b645-990ed51e93ae \u003e DN.FE.ExecuteQuery/865c926e-620f-4107-9fd8-43ac9ee494b3)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[00:15:08.703]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:15:04.4342198Z);\nlet _endTime = datetime(2026-08-25T00:15:04.4342235Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-jk7hknvqffmd'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:15:08.6768100Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;2c78ad4b-fe52-4282-9d01-7f8d1f03d199\",\r\n \"activityId\": \"a7bce1a5-15e0-4dd5-9933-19a9ff7b6f19\",\r\n \"subActivityId\": \"dda68eb2-c740-498f-b62e-c74a25b621f7\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"4d4ca23c-dcca-4b88-9136-446102508ccb\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;2c78ad4b-fe52-4282-9d01-7f8d1f03d199 ARID=a7bce1a5-15e0-4dd5-9933-19a9ff7b6f19 \u003e GW.Http.CallContext/a7bce1a5-15e0-4dd5-9933-19a9ff7b6f19 \u003e GW.ExecuteQuery/4d4ca23c-dcca-4b88-9136-446102508ccb \u003e DN.FE.ExecuteQuery/dda68eb2-c740-498f-b62e-c74a25b621f7)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[00:15:08.705]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "context canceled", "name": "cosmosResourceSnapshots" }�[0m �[37m[00:15:09.096]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:15:04.4342198Z);\nlet _endTime = datetime(2026-08-25T00:15:04.4342235Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-jk7hknvqffmd'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:15:08.6768100Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;2c78ad4b-fe52-4282-9d01-7f8d1f03d199\",\r\n \"activityId\": \"a7bce1a5-15e0-4dd5-9933-19a9ff7b6f19\",\r\n \"subActivityId\": \"dda68eb2-c740-498f-b62e-c74a25b621f7\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"4d4ca23c-dcca-4b88-9136-446102508ccb\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;2c78ad4b-fe52-4282-9d01-7f8d1f03d199 ARID=a7bce1a5-15e0-4dd5-9933-19a9ff7b6f19 \u003e GW.Http.CallContext/a7bce1a5-15e0-4dd5-9933-19a9ff7b6f19 \u003e GW.ExecuteQuery/4d4ca23c-dcca-4b88-9136-446102508ccb \u003e DN.FE.ExecuteQuery/dda68eb2-c740-498f-b62e-c74a25b621f7)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.003s. Expected success, but got an error: <errors.errorString \| 0xc000626a40>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[00:14:56.798]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:14:53.4068554Z);\nlet _endTime = datetime(2026-08-25T00:14:53.4068559Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-jk7hknvqffmd'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:14:56.7779164Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;cb2874aa-23b9-4e90-82b8-8dbdc8ccb12d\",\r\n \"activityId\": \"697030e7-6169-48d4-afab-e55af5423283\",\r\n \"subActivityId\": \"7025f3d5-21ad-4b66-b1ec-6016c4d8ba2f\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"e5629797-dc74-431c-89f2-26cc2ad935e1\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;cb2874aa-23b9-4e90-82b8-8dbdc8ccb12d ARID=697030e7-6169-48d4-afab-e55af5423283 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/d94d70a1-2ca3-4462-b247-1c225640a5b0 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/d94d70a1-2ca3-4462-b247-1c225640a5b0 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/e5629797-dc74-431c-89f2-26cc2ad935e1 \u003e DN.FE.ExecuteQuery/7025f3d5-21ad-4b66-b1ec-6016c4d8ba2f)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[00:14:56.846]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:14:53.4068554Z);\nlet _endTime = datetime(2026-08-25T00:14:53.4068559Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-jk7hknvqffmd'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:14:56.7779164Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;cb2874aa-23b9-4e90-82b8-8dbdc8ccb12d\",\r\n \"activityId\": \"697030e7-6169-48d4-afab-e55af5423283\",\r\n \"subActivityId\": \"7025f3d5-21ad-4b66-b1ec-6016c4d8ba2f\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"e5629797-dc74-431c-89f2-26cc2ad935e1\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;cb2874aa-23b9-4e90-82b8-8dbdc8ccb12d ARID=697030e7-6169-48d4-afab-e55af5423283 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/d94d70a1-2ca3-4462-b247-1c225640a5b0 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/d94d70a1-2ca3-4462-b247-1c225640a5b0 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/e5629797-dc74-431c-89f2-26cc2ad935e1 \u003e DN.FE.ExecuteQuery/7025f3d5-21ad-4b66-b1ec-6016c4d8ba2f)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[00:15:00.265]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:14:56.9537642Z);\nlet _endTime = datetime(2026-08-25T00:14:56.9537648Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-jk7hknvqffmd'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:15:00.2458657Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;d3045808-57d8-4c23-8f73-affb30f6a130\",\r\n \"activityId\": \"60865211-a5af-4242-8bd8-14b7a1f199ba\",\r\n \"subActivityId\": \"865c926e-620f-4107-9fd8-43ac9ee494b3\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"70a56a06-7a14-4b6c-b645-990ed51e93ae\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;d3045808-57d8-4c23-8f73-affb30f6a130 ARID=60865211-a5af-4242-8bd8-14b7a1f199ba \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/230b95d6-3e5a-4f20-bb04-f8704206f9bc \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/230b95d6-3e5a-4f20-bb04-f8704206f9bc \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/70a56a06-7a14-4b6c-b645-990ed51e93ae \u003e DN.FE.ExecuteQuery/865c926e-620f-4107-9fd8-43ac9ee494b3)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[00:15:00.327]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:14:56.9537642Z);\nlet _endTime = datetime(2026-08-25T00:14:56.9537648Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-jk7hknvqffmd'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:15:00.2458657Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;d3045808-57d8-4c23-8f73-affb30f6a130\",\r\n \"activityId\": \"60865211-a5af-4242-8bd8-14b7a1f199ba\",\r\n \"subActivityId\": \"865c926e-620f-4107-9fd8-43ac9ee494b3\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"70a56a06-7a14-4b6c-b645-990ed51e93ae\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;d3045808-57d8-4c23-8f73-affb30f6a130 ARID=60865211-a5af-4242-8bd8-14b7a1f199ba \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/230b95d6-3e5a-4f20-bb04-f8704206f9bc \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/230b95d6-3e5a-4f20-bb04-f8704206f9bc \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/70a56a06-7a14-4b6c-b645-990ed51e93ae \u003e DN.FE.ExecuteQuery/865c926e-620f-4107-9fd8-43ac9ee494b3)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[00:15:08.703]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:15:04.4342198Z);\nlet _endTime = datetime(2026-08-25T00:15:04.4342235Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-jk7hknvqffmd'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:15:08.6768100Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;2c78ad4b-fe52-4282-9d01-7f8d1f03d199\",\r\n \"activityId\": \"a7bce1a5-15e0-4dd5-9933-19a9ff7b6f19\",\r\n \"subActivityId\": \"dda68eb2-c740-498f-b62e-c74a25b621f7\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"4d4ca23c-dcca-4b88-9136-446102508ccb\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;2c78ad4b-fe52-4282-9d01-7f8d1f03d199 ARID=a7bce1a5-15e0-4dd5-9933-19a9ff7b6f19 \u003e GW.Http.CallContext/a7bce1a5-15e0-4dd5-9933-19a9ff7b6f19 \u003e GW.ExecuteQuery/4d4ca23c-dcca-4b88-9136-446102508ccb \u003e DN.FE.ExecuteQuery/dda68eb2-c740-498f-b62e-c74a25b621f7)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[00:15:08.705]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "context canceled", "name": "cosmosResourceSnapshots" }�[0m �[37m[00:15:09.096]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:15:04.4342198Z);\nlet _endTime = datetime(2026-08-25T00:15:04.4342235Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-jk7hknvqffmd'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:15:08.6768100Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;2c78ad4b-fe52-4282-9d01-7f8d1f03d199\",\r\n \"activityId\": \"a7bce1a5-15e0-4dd5-9933-19a9ff7b6f19\",\r\n \"subActivityId\": \"dda68eb2-c740-498f-b62e-c74a25b621f7\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"4d4ca23c-dcca-4b88-9136-446102508ccb\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;2c78ad4b-fe52-4282-9d01-7f8d1f03d199 ARID=a7bce1a5-15e0-4dd5-9933-19a9ff7b6f19 \u003e GW.Http.CallContext/a7bce1a5-15e0-4dd5-9933-19a9ff7b6f19 \u003e GW.ExecuteQuery/4d4ca23c-dcca-4b88-9136-446102508ccb \u003e DN.FE.ExecuteQuery/dda68eb2-c740-498f-b62e-c74a25b621f7)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ...
23:06:53	pull-ci-Azure-ARO-HCP-main-e2e-parallel head af0ab1f	e2e	failed	★#6504 (open)	1	e2e (1) query-basic: command failed: exit status 1 23:06:53 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.006s. Expected success, but got an error: <errors.errorString \| 0xc001f4a670>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[00:15:10.975]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:15:05.9070266Z);\nlet _endTime = datetime(2026-08-25T00:15:05.9070271Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hl4wgvvqgzxr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:15:10.9519459Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;26675931-f7e7-4344-9b85-24a650b39392\",\r\n \"activityId\": \"ef76707e-b3d2-4ccf-b2b7-9437b788ee47\",\r\n \"subActivityId\": \"d1814cb1-6831-44dc-8356-c144fb092d85\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"ee35d653-7145-4b4c-9e8d-2c7d4c3f2774\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;26675931-f7e7-4344-9b85-24a650b39392 ARID=ef76707e-b3d2-4ccf-b2b7-9437b788ee47 \u003e GW.Http.CallContext/ef76707e-b3d2-4ccf-b2b7-9437b788ee47 \u003e GW.ExecuteQuery/ee35d653-7145-4b4c-9e8d-2c7d4c3f2774 \u003e DN.FE.ExecuteQuery/d1814cb1-6831-44dc-8356-c144fb092d85)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[00:15:11.171]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:15:05.9070266Z);\nlet _endTime = datetime(2026-08-25T00:15:05.9070271Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hl4wgvvqgzxr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:15:10.9519459Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;26675931-f7e7-4344-9b85-24a650b39392\",\r\n \"activityId\": \"ef76707e-b3d2-4ccf-b2b7-9437b788ee47\",\r\n \"subActivityId\": \"d1814cb1-6831-44dc-8356-c144fb092d85\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"ee35d653-7145-4b4c-9e8d-2c7d4c3f2774\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;26675931-f7e7-4344-9b85-24a650b39392 ARID=ef76707e-b3d2-4ccf-b2b7-9437b788ee47 \u003e GW.Http.CallContext/ef76707e-b3d2-4ccf-b2b7-9437b788ee47 \u003e GW.ExecuteQuery/ee35d653-7145-4b4c-9e8d-2c7d4c3f2774 \u003e DN.FE.ExecuteQuery/d1814cb1-6831-44dc-8356-c144fb092d85)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[00:15:15.030]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:15:11.2948478Z);\nlet _endTime = datetime(2026-08-25T00:15:11.2948504Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hl4wgvvqgzxr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:15:15.0090681Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;b3257560-3cb6-4fcc-8e7f-ed9367adc845\",\r\n \"activityId\": \"c0adf2ff-92d7-4876-9381-3d81d9c09dcf\",\r\n \"subActivityId\": \"d36c0b97-879a-48b5-9161-c4d48dd18d09\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"a014e706-b98c-4023-88e5-6de86b03b1af\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;b3257560-3cb6-4fcc-8e7f-ed9367adc845 ARID=c0adf2ff-92d7-4876-9381-3d81d9c09dcf \u003e GW.Http.CallContext/c0adf2ff-92d7-4876-9381-3d81d9c09dcf \u003e GW.ExecuteQuery/a014e706-b98c-4023-88e5-6de86b03b1af \u003e DN.FE.ExecuteQuery/d36c0b97-879a-48b5-9161-c4d48dd18d09)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[00:15:15.124]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:15:11.2948478Z);\nlet _endTime = datetime(2026-08-25T00:15:11.2948504Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hl4wgvvqgzxr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:15:15.0090681Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;b3257560-3cb6-4fcc-8e7f-ed9367adc845\",\r\n \"activityId\": \"c0adf2ff-92d7-4876-9381-3d81d9c09dcf\",\r\n \"subActivityId\": \"d36c0b97-879a-48b5-9161-c4d48dd18d09\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"a014e706-b98c-4023-88e5-6de86b03b1af\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;b3257560-3cb6-4fcc-8e7f-ed9367adc845 ARID=c0adf2ff-92d7-4876-9381-3d81d9c09dcf \u003e GW.Http.CallContext/c0adf2ff-92d7-4876-9381-3d81d9c09dcf \u003e GW.ExecuteQuery/a014e706-b98c-4023-88e5-6de86b03b1af \u003e DN.FE.ExecuteQuery/d36c0b97-879a-48b5-9161-c4d48dd18d09)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[00:15:22.332]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:15:19.0004961Z);\nlet _endTime = datetime(2026-08-25T00:15:19.0004966Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hl4wgvvqgzxr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:15:22.3098397Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;b394207c-245e-45d5-a804-f89ebe5bae42\",\r\n \"activityId\": \"00780658-87c4-4f5e-a63b-4d7f35817d45\",\r\n \"subActivityId\": \"08e72ece-2adb-4757-8239-b00748f8c803\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"4e923dde-12b5-4490-b57b-833a13d432e8\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;b394207c-245e-45d5-a804-f89ebe5bae42 ARID=00780658-87c4-4f5e-a63b-4d7f35817d45 \u003e GW.Http.CallContext/00780658-87c4-4f5e-a63b-4d7f35817d45 \u003e GW.ExecuteQuery/4e923dde-12b5-4490-b57b-833a13d432e8 \u003e DN.FE.ExecuteQuery/08e72ece-2adb-4757-8239-b00748f8c803)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[00:15:22.436]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:15:19.0004961Z);\nlet _endTime = datetime(2026-08-25T00:15:19.0004966Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hl4wgvvqgzxr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:15:22.3098397Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;b394207c-245e-45d5-a804-f89ebe5bae42\",\r\n \"activityId\": \"00780658-87c4-4f5e-a63b-4d7f35817d45\",\r\n \"subActivityId\": \"08e72ece-2adb-4757-8239-b00748f8c803\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"4e923dde-12b5-4490-b57b-833a13d432e8\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;b394207c-245e-45d5-a804-f89ebe5bae42 ARID=00780658-87c4-4f5e-a63b-4d7f35817d45 \u003e GW.Http.CallContext/00780658-87c4-4f5e-a63b-4d7f35817d45 \u003e GW.ExecuteQuery/4e923dde-12b5-4490-b57b-833a13d432e8 \u003e DN.FE.ExecuteQuery/08e72ece-2adb-4757-8239-b00748f8c803)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.006s. Expected success, but got an error: <errors.errorString \| 0xc001f4a670>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[00:15:10.975]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:15:05.9070266Z);\nlet _endTime = datetime(2026-08-25T00:15:05.9070271Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hl4wgvvqgzxr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:15:10.9519459Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;26675931-f7e7-4344-9b85-24a650b39392\",\r\n \"activityId\": \"ef76707e-b3d2-4ccf-b2b7-9437b788ee47\",\r\n \"subActivityId\": \"d1814cb1-6831-44dc-8356-c144fb092d85\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"ee35d653-7145-4b4c-9e8d-2c7d4c3f2774\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;26675931-f7e7-4344-9b85-24a650b39392 ARID=ef76707e-b3d2-4ccf-b2b7-9437b788ee47 \u003e GW.Http.CallContext/ef76707e-b3d2-4ccf-b2b7-9437b788ee47 \u003e GW.ExecuteQuery/ee35d653-7145-4b4c-9e8d-2c7d4c3f2774 \u003e DN.FE.ExecuteQuery/d1814cb1-6831-44dc-8356-c144fb092d85)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[00:15:11.171]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:15:05.9070266Z);\nlet _endTime = datetime(2026-08-25T00:15:05.9070271Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hl4wgvvqgzxr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:15:10.9519459Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;26675931-f7e7-4344-9b85-24a650b39392\",\r\n \"activityId\": \"ef76707e-b3d2-4ccf-b2b7-9437b788ee47\",\r\n \"subActivityId\": \"d1814cb1-6831-44dc-8356-c144fb092d85\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"ee35d653-7145-4b4c-9e8d-2c7d4c3f2774\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;26675931-f7e7-4344-9b85-24a650b39392 ARID=ef76707e-b3d2-4ccf-b2b7-9437b788ee47 \u003e GW.Http.CallContext/ef76707e-b3d2-4ccf-b2b7-9437b788ee47 \u003e GW.ExecuteQuery/ee35d653-7145-4b4c-9e8d-2c7d4c3f2774 \u003e DN.FE.ExecuteQuery/d1814cb1-6831-44dc-8356-c144fb092d85)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[00:15:15.030]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:15:11.2948478Z);\nlet _endTime = datetime(2026-08-25T00:15:11.2948504Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hl4wgvvqgzxr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:15:15.0090681Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;b3257560-3cb6-4fcc-8e7f-ed9367adc845\",\r\n \"activityId\": \"c0adf2ff-92d7-4876-9381-3d81d9c09dcf\",\r\n \"subActivityId\": \"d36c0b97-879a-48b5-9161-c4d48dd18d09\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"a014e706-b98c-4023-88e5-6de86b03b1af\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;b3257560-3cb6-4fcc-8e7f-ed9367adc845 ARID=c0adf2ff-92d7-4876-9381-3d81d9c09dcf \u003e GW.Http.CallContext/c0adf2ff-92d7-4876-9381-3d81d9c09dcf \u003e GW.ExecuteQuery/a014e706-b98c-4023-88e5-6de86b03b1af \u003e DN.FE.ExecuteQuery/d36c0b97-879a-48b5-9161-c4d48dd18d09)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[00:15:15.124]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:15:11.2948478Z);\nlet _endTime = datetime(2026-08-25T00:15:11.2948504Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hl4wgvvqgzxr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:15:15.0090681Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;b3257560-3cb6-4fcc-8e7f-ed9367adc845\",\r\n \"activityId\": \"c0adf2ff-92d7-4876-9381-3d81d9c09dcf\",\r\n \"subActivityId\": \"d36c0b97-879a-48b5-9161-c4d48dd18d09\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"a014e706-b98c-4023-88e5-6de86b03b1af\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;b3257560-3cb6-4fcc-8e7f-ed9367adc845 ARID=c0adf2ff-92d7-4876-9381-3d81d9c09dcf \u003e GW.Http.CallContext/c0adf2ff-92d7-4876-9381-3d81d9c09dcf \u003e GW.ExecuteQuery/a014e706-b98c-4023-88e5-6de86b03b1af \u003e DN.FE.ExecuteQuery/d36c0b97-879a-48b5-9161-c4d48dd18d09)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[00:15:22.332]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:15:19.0004961Z);\nlet _endTime = datetime(2026-08-25T00:15:19.0004966Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hl4wgvvqgzxr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:15:22.3098397Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;b394207c-245e-45d5-a804-f89ebe5bae42\",\r\n \"activityId\": \"00780658-87c4-4f5e-a63b-4d7f35817d45\",\r\n \"subActivityId\": \"08e72ece-2adb-4757-8239-b00748f8c803\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"4e923dde-12b5-4490-b57b-833a13d432e8\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;b394207c-245e-45d5-a804-f89ebe5bae42 ARID=00780658-87c4-4f5e-a63b-4d7f35817d45 \u003e GW.Http.CallContext/00780658-87c4-4f5e-a63b-4d7f35817d45 \u003e GW.ExecuteQuery/4e923dde-12b5-4490-b57b-833a13d432e8 \u003e DN.FE.ExecuteQuery/08e72ece-2adb-4757-8239-b00748f8c803)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[00:15:22.436]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:15:19.0004961Z);\nlet _endTime = datetime(2026-08-25T00:15:19.0004966Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hl4wgvvqgzxr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:15:22.3098397Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;b394207c-245e-45d5-a804-f89ebe5bae42\",\r\n \"activityId\": \"00780658-87c4-4f5e-a63b-4d7f35817d45\",\r\n \"subActivityId\": \"08e72ece-2adb-4757-8239-b00748f8c803\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"4e923dde-12b5-4490-b57b-833a13d432e8\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;b394207c-245e-45d5-a804-f89ebe5bae42 ARID=00780658-87c4-4f5e-a63b-4d7f35817d45 \u003e GW.Http.CallContext/00780658-87c4-4f5e-a63b-4d7f35817d45 \u003e GW.ExecuteQuery/4e923dde-12b5-4490-b57b-833a13d432e8 \u003e DN.FE.ExecuteQuery/08e72ece-2adb-4757-8239-b00748f8c803)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ...
22:44:28	pull-ci-Azure-ARO-HCP-main-e2e-parallel head aa2fb7c	alert, e2e	failed	★#6635 (open)	3	e2e (2) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 22:44:28 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-x45vwlgpgvzk/cluster-candidate-4-21-rztpw4 should provision Unexpected error: <fmt.wrapError \| 0xc000f3e6a0>: failed to create HCP cluster cluster-candidate-4-21-rztpw4: failed waiting for cluster="cluster-candidate-4-21-rztpw4" in resourcegroup="rg-candidate-4-21-x45vwlgpgvzk" to finish creating: GET https://rp.j4709760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e986c420-0503-4186-9a01-bad1f92a005c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e986c420-0503-4186-9a01-bad1f92a005c", "name": "e986c420-0503-4186-9a01-bad1f92a005c", "status": "Failed", "startTime": "2026-08-24T23:37:07.202301694Z", "endTime": "2026-08-24T23:56:10.677598487Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-x45vwlgpgvzk/cluster-candidate-4-21-rztpw4 should provision Unexpected error: <fmt.wrapError \| 0xc000f3e6a0>: failed to create HCP cluster cluster-candidate-4-21-rztpw4: failed waiting for cluster="cluster-candidate-4-21-rztpw4" in resourcegroup="rg-candidate-4-21-x45vwlgpgvzk" to finish creating: GET https://rp.j4709760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e986c420-0503-4186-9a01-bad1f92a005c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e986c420-0503-4186-9a01-bad1f92a005c", "name": "e986c420-0503-4186-9a01-bad1f92a005c", "status": "Failed", "startTime": "2026-08-24T23:37:07.202301694Z", "endTime": "2026-08-24T23:56:10.677598487Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred query-basic: command failed: exit status 1 22:44:28 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.004s. Expected success, but got an error: <errors.errorString \| 0xc0007c8410>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[23:59:56.120]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:59:52.8342332Z);\nlet _endTime = datetime(2026-08-24T23:59:52.8342339Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6pmwg8htdhg9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:59:56.0986998Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;589862cc-bbb1-4912-808c-32c7239931d3\",\r\n \"activityId\": \"25c493b7-b925-4f70-a720-29da9868f2ec\",\r\n \"subActivityId\": \"77734989-d965-4eea-ad4e-90e589bfc994\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"03f18cdd-a350-4699-9b6f-326a47f2842c\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;589862cc-bbb1-4912-808c-32c7239931d3 ARID=25c493b7-b925-4f70-a720-29da9868f2ec \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/4bbda136-b57d-4b2b-810c-8cb7b1bbdba1 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/4bbda136-b57d-4b2b-810c-8cb7b1bbdba1 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/03f18cdd-a350-4699-9b6f-326a47f2842c \u003e DN.FE.ExecuteQuery/77734989-d965-4eea-ad4e-90e589bfc994)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[23:59:56.169]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:59:52.8342332Z);\nlet _endTime = datetime(2026-08-24T23:59:52.8342339Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6pmwg8htdhg9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:59:56.0986998Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;589862cc-bbb1-4912-808c-32c7239931d3\",\r\n \"activityId\": \"25c493b7-b925-4f70-a720-29da9868f2ec\",\r\n \"subActivityId\": \"77734989-d965-4eea-ad4e-90e589bfc994\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"03f18cdd-a350-4699-9b6f-326a47f2842c\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;589862cc-bbb1-4912-808c-32c7239931d3 ARID=25c493b7-b925-4f70-a720-29da9868f2ec \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/4bbda136-b57d-4b2b-810c-8cb7b1bbdba1 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/4bbda136-b57d-4b2b-810c-8cb7b1bbdba1 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/03f18cdd-a350-4699-9b6f-326a47f2842c \u003e DN.FE.ExecuteQuery/77734989-d965-4eea-ad4e-90e589bfc994)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[23:59:59.562]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:59:56.2555369Z);\nlet _endTime = datetime(2026-08-24T23:59:56.2555377Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6pmwg8htdhg9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:59:59.5412585Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;8bde4e19-6492-4fdd-8666-87105fc1bc29\",\r\n \"activityId\": \"2dd85a88-1f1b-4ca3-a627-624434e67a6e\",\r\n \"subActivityId\": \"7321b56c-2660-40dd-827d-5978e678ae28\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"b051aa4d-89e0-4106-a0dd-7ed70662f0f5\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;8bde4e19-6492-4fdd-8666-87105fc1bc29 ARID=2dd85a88-1f1b-4ca3-a627-624434e67a6e \u003e GW.Http.CallContext/2dd85a88-1f1b-4ca3-a627-624434e67a6e \u003e GW.ExecuteQuery/b051aa4d-89e0-4106-a0dd-7ed70662f0f5 \u003e DN.FE.ExecuteQuery/7321b56c-2660-40dd-827d-5978e678ae28)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[23:59:59.611]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:59:56.2555369Z);\nlet _endTime = datetime(2026-08-24T23:59:56.2555377Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6pmwg8htdhg9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:59:59.5412585Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;8bde4e19-6492-4fdd-8666-87105fc1bc29\",\r\n \"activityId\": \"2dd85a88-1f1b-4ca3-a627-624434e67a6e\",\r\n \"subActivityId\": \"7321b56c-2660-40dd-827d-5978e678ae28\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"b051aa4d-89e0-4106-a0dd-7ed70662f0f5\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;8bde4e19-6492-4fdd-8666-87105fc1bc29 ARID=2dd85a88-1f1b-4ca3-a627-624434e67a6e \u003e GW.Http.CallContext/2dd85a88-1f1b-4ca3-a627-624434e67a6e \u003e GW.ExecuteQuery/b051aa4d-89e0-4106-a0dd-7ed70662f0f5 \u003e DN.FE.ExecuteQuery/7321b56c-2660-40dd-827d-5978e678ae28)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[00:00:06.688]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:00:03.1874189Z);\nlet _endTime = datetime(2026-08-25T00:00:03.1874200Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6pmwg8htdhg9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:00:06.6685554Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;8161fae6-5ba5-437e-9969-ded5f2fa4099\",\r\n \"activityId\": \"f6ea14a6-45ea-406a-84a9-e4660aa2592c\",\r\n \"subActivityId\": \"a15ff010-b600-4c98-b9b4-eec67c625699\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"95b62cbe-f648-4de9-8ee1-66a05ab19238\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;8161fae6-5ba5-437e-9969-ded5f2fa4099 ARID=f6ea14a6-45ea-406a-84a9-e4660aa2592c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/4a46171d-4922-4ebe-a2fb-926ed44edad6 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/4a46171d-4922-4ebe-a2fb-926ed44edad6 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/95b62cbe-f648-4de9-8ee1-66a05ab19238 \u003e DN.FE.ExecuteQuery/a15ff010-b600-4c98-b9b4-eec67c625699)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[00:00:06.794]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:00:03.1874189Z);\nlet _endTime = datetime(2026-08-25T00:00:03.1874200Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6pmwg8htdhg9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:00:06.6685554Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;8161fae6-5ba5-437e-9969-ded5f2fa4099\",\r\n \"activityId\": \"f6ea14a6-45ea-406a-84a9-e4660aa2592c\",\r\n \"subActivityId\": \"a15ff010-b600-4c98-b9b4-eec67c625699\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"95b62cbe-f648-4de9-8ee1-66a05ab19238\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;8161fae6-5ba5-437e-9969-ded5f2fa4099 ARID=f6ea14a6-45ea-406a-84a9-e4660aa2592c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/4a46171d-4922-4ebe-a2fb-926ed44edad6 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/4a46171d-4922-4ebe-a2fb-926ed44edad6 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/95b62cbe-f648-4de9-8ee1-66a05ab19238 \u003e DN.FE.ExecuteQuery/a15ff010-b600-4c98-b9b4-eec67c625699)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.004s. Expected success, but got an error: <errors.errorString \| 0xc0007c8410>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[23:59:56.120]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:59:52.8342332Z);\nlet _endTime = datetime(2026-08-24T23:59:52.8342339Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6pmwg8htdhg9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:59:56.0986998Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;589862cc-bbb1-4912-808c-32c7239931d3\",\r\n \"activityId\": \"25c493b7-b925-4f70-a720-29da9868f2ec\",\r\n \"subActivityId\": \"77734989-d965-4eea-ad4e-90e589bfc994\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"03f18cdd-a350-4699-9b6f-326a47f2842c\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;589862cc-bbb1-4912-808c-32c7239931d3 ARID=25c493b7-b925-4f70-a720-29da9868f2ec \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/4bbda136-b57d-4b2b-810c-8cb7b1bbdba1 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/4bbda136-b57d-4b2b-810c-8cb7b1bbdba1 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/03f18cdd-a350-4699-9b6f-326a47f2842c \u003e DN.FE.ExecuteQuery/77734989-d965-4eea-ad4e-90e589bfc994)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[23:59:56.169]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:59:52.8342332Z);\nlet _endTime = datetime(2026-08-24T23:59:52.8342339Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6pmwg8htdhg9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:59:56.0986998Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;589862cc-bbb1-4912-808c-32c7239931d3\",\r\n \"activityId\": \"25c493b7-b925-4f70-a720-29da9868f2ec\",\r\n \"subActivityId\": \"77734989-d965-4eea-ad4e-90e589bfc994\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"03f18cdd-a350-4699-9b6f-326a47f2842c\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;589862cc-bbb1-4912-808c-32c7239931d3 ARID=25c493b7-b925-4f70-a720-29da9868f2ec \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/4bbda136-b57d-4b2b-810c-8cb7b1bbdba1 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/4bbda136-b57d-4b2b-810c-8cb7b1bbdba1 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/03f18cdd-a350-4699-9b6f-326a47f2842c \u003e DN.FE.ExecuteQuery/77734989-d965-4eea-ad4e-90e589bfc994)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[23:59:59.562]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:59:56.2555369Z);\nlet _endTime = datetime(2026-08-24T23:59:56.2555377Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6pmwg8htdhg9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:59:59.5412585Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;8bde4e19-6492-4fdd-8666-87105fc1bc29\",\r\n \"activityId\": \"2dd85a88-1f1b-4ca3-a627-624434e67a6e\",\r\n \"subActivityId\": \"7321b56c-2660-40dd-827d-5978e678ae28\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"b051aa4d-89e0-4106-a0dd-7ed70662f0f5\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;8bde4e19-6492-4fdd-8666-87105fc1bc29 ARID=2dd85a88-1f1b-4ca3-a627-624434e67a6e \u003e GW.Http.CallContext/2dd85a88-1f1b-4ca3-a627-624434e67a6e \u003e GW.ExecuteQuery/b051aa4d-89e0-4106-a0dd-7ed70662f0f5 \u003e DN.FE.ExecuteQuery/7321b56c-2660-40dd-827d-5978e678ae28)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[23:59:59.611]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:59:56.2555369Z);\nlet _endTime = datetime(2026-08-24T23:59:56.2555377Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6pmwg8htdhg9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:59:59.5412585Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;8bde4e19-6492-4fdd-8666-87105fc1bc29\",\r\n \"activityId\": \"2dd85a88-1f1b-4ca3-a627-624434e67a6e\",\r\n \"subActivityId\": \"7321b56c-2660-40dd-827d-5978e678ae28\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"b051aa4d-89e0-4106-a0dd-7ed70662f0f5\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;8bde4e19-6492-4fdd-8666-87105fc1bc29 ARID=2dd85a88-1f1b-4ca3-a627-624434e67a6e \u003e GW.Http.CallContext/2dd85a88-1f1b-4ca3-a627-624434e67a6e \u003e GW.ExecuteQuery/b051aa4d-89e0-4106-a0dd-7ed70662f0f5 \u003e DN.FE.ExecuteQuery/7321b56c-2660-40dd-827d-5978e678ae28)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[00:00:06.688]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:00:03.1874189Z);\nlet _endTime = datetime(2026-08-25T00:00:03.1874200Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6pmwg8htdhg9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:00:06.6685554Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;8161fae6-5ba5-437e-9969-ded5f2fa4099\",\r\n \"activityId\": \"f6ea14a6-45ea-406a-84a9-e4660aa2592c\",\r\n \"subActivityId\": \"a15ff010-b600-4c98-b9b4-eec67c625699\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"95b62cbe-f648-4de9-8ee1-66a05ab19238\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;8161fae6-5ba5-437e-9969-ded5f2fa4099 ARID=f6ea14a6-45ea-406a-84a9-e4660aa2592c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/4a46171d-4922-4ebe-a2fb-926ed44edad6 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/4a46171d-4922-4ebe-a2fb-926ed44edad6 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/95b62cbe-f648-4de9-8ee1-66a05ab19238 \u003e DN.FE.ExecuteQuery/a15ff010-b600-4c98-b9b4-eec67c625699)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[00:00:06.794]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T00:00:03.1874189Z);\nlet _endTime = datetime(2026-08-25T00:00:03.1874200Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6pmwg8htdhg9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T00:00:06.6685554Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;8161fae6-5ba5-437e-9969-ded5f2fa4099\",\r\n \"activityId\": \"f6ea14a6-45ea-406a-84a9-e4660aa2592c\",\r\n \"subActivityId\": \"a15ff010-b600-4c98-b9b4-eec67c625699\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"95b62cbe-f648-4de9-8ee1-66a05ab19238\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;8161fae6-5ba5-437e-9969-ded5f2fa4099 ARID=f6ea14a6-45ea-406a-84a9-e4660aa2592c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/4a46171d-4922-4ebe-a2fb-926ed44edad6 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/4a46171d-4922-4ebe-a2fb-926ed44edad6 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/95b62cbe-f648-4de9-8ee1-66a05ab19238 \u003e DN.FE.ExecuteQuery/a15ff010-b600-4c98-b9b4-eec67c625699)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... alert (1) alert [hcp] KubePodNotReady fired 22:44:28 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-24T23:47:43Z Ended: 2026-08-25T00:09:40Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j4709760-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd8f93h71j1qrcah2q03eja41gojmif-i6w1a1c6f6m8f7l", pod="router-7f97cf9c5-phjp6", severity="warning" Description: Pod ocm-arohcpci01-2sd8f93h71j1qrcah2q03eja41gojmif-i6w1a1c6f6m8f7l/router-7f97cf9c5-phjp6 has been in a non-ready state for longer than 5 minutes.
22:40:47	pull-ci-Azure-ARO-HCP-main-e2e-parallel head e8abf5d	e2e	failed	#6663 (open)	2	e2e (2) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 22:40:47 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc000d48840>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-mglhphvdtjhd" to finish creating: GET https://rp.j6339200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1dbfdf90-d2f5-4efc-b68b-42b49d8220eb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1dbfdf90-d2f5-4efc-b68b-42b49d8220eb", "name": "1dbfdf90-d2f5-4efc-b68b-42b49d8220eb", "status": "Failed", "startTime": "2026-08-24T23:29:18.405302537Z", "endTime": "2026-08-24T23:48:26.458954406Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc000d48840>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-mglhphvdtjhd" to finish creating: GET https://rp.j6339200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1dbfdf90-d2f5-4efc-b68b-42b49d8220eb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1dbfdf90-d2f5-4efc-b68b-42b49d8220eb", "name": "1dbfdf90-d2f5-4efc-b68b-42b49d8220eb", "status": "Failed", "startTime": "2026-08-24T23:29:18.405302537Z", "endTime": "2026-08-24T23:48:26.458954406Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 22:40:47 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc0010a6520>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-ztqxhwgzpbxg" to finish creating: GET https://rp.j6339200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a8cd452b-9528-42a8-b489-fc4c1edf89ab -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a8cd452b-9528-42a8-b489-fc4c1edf89ab", "name": "a8cd452b-9528-42a8-b489-fc4c1edf89ab", "status": "Failed", "startTime": "2026-08-24T23:28:24.136815957Z", "endTime": "2026-08-24T23:47:26.678538436Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc0010a6520>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-ztqxhwgzpbxg" to finish creating: GET https://rp.j6339200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a8cd452b-9528-42a8-b489-fc4c1edf89ab -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a8cd452b-9528-42a8-b489-fc4c1edf89ab", "name": "a8cd452b-9528-42a8-b489-fc4c1edf89ab", "status": "Failed", "startTime": "2026-08-24T23:28:24.136815957Z", "endTime": "2026-08-24T23:47:26.678538436Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred
22:28:33	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 86bc57f	e2e	failed	★#6658 (open)	3	e2e (3) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 22:28:33 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc001404860>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-4vpnfc6cs6l8" to finish creating: GET https://rp.j4721408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/56cea9aa-e8d5-4c87-9934-6423e81ae186 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/56cea9aa-e8d5-4c87-9934-6423e81ae186", "name": "56cea9aa-e8d5-4c87-9934-6423e81ae186", "status": "Failed", "startTime": "2026-08-24T23:13:22.86871269Z", "endTime": "2026-08-24T23:32:23.316177558Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc001404860>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-4vpnfc6cs6l8" to finish creating: GET https://rp.j4721408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/56cea9aa-e8d5-4c87-9934-6423e81ae186 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/56cea9aa-e8d5-4c87-9934-6423e81ae186", "name": "56cea9aa-e8d5-4c87-9934-6423e81ae186", "status": "Failed", "startTime": "2026-08-24T23:13:22.86871269Z", "endTime": "2026-08-24T23:32:23.316177558Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 22:28:33 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc000e90140>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-zmfzwkvx7j5m" to finish creating: GET https://rp.j4721408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0f0de8f-0832-4d52-8107-854a84528fff -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0f0de8f-0832-4d52-8107-854a84528fff", "name": "f0f0de8f-0832-4d52-8107-854a84528fff", "status": "Failed", "startTime": "2026-08-24T23:13:10.586040355Z", "endTime": "2026-08-24T23:32:13.301339783Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc000e90140>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-zmfzwkvx7j5m" to finish creating: GET https://rp.j4721408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0f0de8f-0832-4d52-8107-854a84528fff -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0f0de8f-0832-4d52-8107-854a84528fff", "name": "f0f0de8f-0832-4d52-8107-854a84528fff", "status": "Failed", "startTime": "2026-08-24T23:13:10.586040355Z", "endTime": "2026-08-24T23:32:13.301339783Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred query-basic: command failed: exit status 1 22:28:33 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.007s. Expected success, but got an error: <errors.errorString \| 0xc000213240>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[23:38:31.458]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:38:28.2685815Z);\nlet _endTime = datetime(2026-08-24T23:38:28.2685822Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-pgv7kxdwgbs9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:38:31.4336345Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;f3cb75f1-d7c9-4b87-adcc-5c7920c1ae76\",\r\n \"activityId\": \"13caf9ec-f590-4f8b-a99f-4bb19faac07c\",\r\n \"subActivityId\": \"789e5af0-c128-4f5a-8973-5ec8f7451bb0\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"5a471082-4419-4989-9a9e-467c6c7c4fd6\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;f3cb75f1-d7c9-4b87-adcc-5c7920c1ae76 ARID=13caf9ec-f590-4f8b-a99f-4bb19faac07c \u003e GW.Http.CallContext/13caf9ec-f590-4f8b-a99f-4bb19faac07c \u003e GW.ExecuteQuery/5a471082-4419-4989-9a9e-467c6c7c4fd6 \u003e DN.FE.ExecuteQuery/789e5af0-c128-4f5a-8973-5ec8f7451bb0)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[23:38:31.517]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:38:28.2685815Z);\nlet _endTime = datetime(2026-08-24T23:38:28.2685822Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-pgv7kxdwgbs9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:38:31.4336345Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;f3cb75f1-d7c9-4b87-adcc-5c7920c1ae76\",\r\n \"activityId\": \"13caf9ec-f590-4f8b-a99f-4bb19faac07c\",\r\n \"subActivityId\": \"789e5af0-c128-4f5a-8973-5ec8f7451bb0\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"5a471082-4419-4989-9a9e-467c6c7c4fd6\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;f3cb75f1-d7c9-4b87-adcc-5c7920c1ae76 ARID=13caf9ec-f590-4f8b-a99f-4bb19faac07c \u003e GW.Http.CallContext/13caf9ec-f590-4f8b-a99f-4bb19faac07c \u003e GW.ExecuteQuery/5a471082-4419-4989-9a9e-467c6c7c4fd6 \u003e DN.FE.ExecuteQuery/789e5af0-c128-4f5a-8973-5ec8f7451bb0)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[23:38:34.673]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:38:31.5847811Z);\nlet _endTime = datetime(2026-08-24T23:38:31.5847819Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-pgv7kxdwgbs9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:38:34.6523649Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;defe9e59-a9e0-410e-9251-a03a8a2d2f6b\",\r\n \"activityId\": \"56e0ac7d-4433-4a86-9cfe-6905af57c05a\",\r\n \"subActivityId\": \"ad63d932-4f9e-4483-9ab8-8ec613cf4164\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"6ed21c2c-224d-44ac-9721-302de763024e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;defe9e59-a9e0-410e-9251-a03a8a2d2f6b ARID=56e0ac7d-4433-4a86-9cfe-6905af57c05a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/4ddc6f31-9ca8-4fa6-acc9-e2c28ed3c59d \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/4ddc6f31-9ca8-4fa6-acc9-e2c28ed3c59d \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/6ed21c2c-224d-44ac-9721-302de763024e \u003e DN.FE.ExecuteQuery/ad63d932-4f9e-4483-9ab8-8ec613cf4164)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[23:38:34.725]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:38:31.5847811Z);\nlet _endTime = datetime(2026-08-24T23:38:31.5847819Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-pgv7kxdwgbs9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:38:34.6523649Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;defe9e59-a9e0-410e-9251-a03a8a2d2f6b\",\r\n \"activityId\": \"56e0ac7d-4433-4a86-9cfe-6905af57c05a\",\r\n \"subActivityId\": \"ad63d932-4f9e-4483-9ab8-8ec613cf4164\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"6ed21c2c-224d-44ac-9721-302de763024e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;defe9e59-a9e0-410e-9251-a03a8a2d2f6b ARID=56e0ac7d-4433-4a86-9cfe-6905af57c05a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/4ddc6f31-9ca8-4fa6-acc9-e2c28ed3c59d \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/4ddc6f31-9ca8-4fa6-acc9-e2c28ed3c59d \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/6ed21c2c-224d-44ac-9721-302de763024e \u003e DN.FE.ExecuteQuery/ad63d932-4f9e-4483-9ab8-8ec613cf4164)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[23:38:51.573]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:38:48.1146150Z);\nlet _endTime = datetime(2026-08-24T23:38:48.1146157Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-pgv7kxdwgbs9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:38:51.5509925Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;54a98755-cda5-4e00-b75a-8a9f4a05e23a\",\r\n \"activityId\": \"07bd916f-8e74-4f33-9544-38554323dccf\",\r\n \"subActivityId\": \"ee30f34f-230c-4bd2-af26-3c459201c20c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"95379035-872f-45ec-8ef7-b6611ac9f3a9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;54a98755-cda5-4e00-b75a-8a9f4a05e23a ARID=07bd916f-8e74-4f33-9544-38554323dccf \u003e GW.Http.CallContext/07bd916f-8e74-4f33-9544-38554323dccf \u003e GW.ExecuteQuery/95379035-872f-45ec-8ef7-b6611ac9f3a9 \u003e DN.FE.ExecuteQuery/ee30f34f-230c-4bd2-af26-3c459201c20c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[23:38:51.682]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:38:48.1146150Z);\nlet _endTime = datetime(2026-08-24T23:38:48.1146157Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-pgv7kxdwgbs9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:38:51.5509925Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;54a98755-cda5-4e00-b75a-8a9f4a05e23a\",\r\n \"activityId\": \"07bd916f-8e74-4f33-9544-38554323dccf\",\r\n \"subActivityId\": \"ee30f34f-230c-4bd2-af26-3c459201c20c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"95379035-872f-45ec-8ef7-b6611ac9f3a9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;54a98755-cda5-4e00-b75a-8a9f4a05e23a ARID=07bd916f-8e74-4f33-9544-38554323dccf \u003e GW.Http.CallContext/07bd916f-8e74-4f33-9544-38554323dccf \u003e GW.ExecuteQuery/95379035-872f-45ec-8ef7-b6611ac9f3a9 \u003e DN.FE.ExecuteQuery/ee30f34f-230c-4bd2-af26-3c459201c20c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.007s. Expected success, but got an error: <errors.errorString \| 0xc000213240>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[23:38:31.458]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:38:28.2685815Z);\nlet _endTime = datetime(2026-08-24T23:38:28.2685822Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-pgv7kxdwgbs9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:38:31.4336345Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;f3cb75f1-d7c9-4b87-adcc-5c7920c1ae76\",\r\n \"activityId\": \"13caf9ec-f590-4f8b-a99f-4bb19faac07c\",\r\n \"subActivityId\": \"789e5af0-c128-4f5a-8973-5ec8f7451bb0\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"5a471082-4419-4989-9a9e-467c6c7c4fd6\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;f3cb75f1-d7c9-4b87-adcc-5c7920c1ae76 ARID=13caf9ec-f590-4f8b-a99f-4bb19faac07c \u003e GW.Http.CallContext/13caf9ec-f590-4f8b-a99f-4bb19faac07c \u003e GW.ExecuteQuery/5a471082-4419-4989-9a9e-467c6c7c4fd6 \u003e DN.FE.ExecuteQuery/789e5af0-c128-4f5a-8973-5ec8f7451bb0)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[23:38:31.517]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:38:28.2685815Z);\nlet _endTime = datetime(2026-08-24T23:38:28.2685822Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-pgv7kxdwgbs9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:38:31.4336345Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;f3cb75f1-d7c9-4b87-adcc-5c7920c1ae76\",\r\n \"activityId\": \"13caf9ec-f590-4f8b-a99f-4bb19faac07c\",\r\n \"subActivityId\": \"789e5af0-c128-4f5a-8973-5ec8f7451bb0\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"5a471082-4419-4989-9a9e-467c6c7c4fd6\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;f3cb75f1-d7c9-4b87-adcc-5c7920c1ae76 ARID=13caf9ec-f590-4f8b-a99f-4bb19faac07c \u003e GW.Http.CallContext/13caf9ec-f590-4f8b-a99f-4bb19faac07c \u003e GW.ExecuteQuery/5a471082-4419-4989-9a9e-467c6c7c4fd6 \u003e DN.FE.ExecuteQuery/789e5af0-c128-4f5a-8973-5ec8f7451bb0)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[23:38:34.673]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:38:31.5847811Z);\nlet _endTime = datetime(2026-08-24T23:38:31.5847819Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-pgv7kxdwgbs9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:38:34.6523649Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;defe9e59-a9e0-410e-9251-a03a8a2d2f6b\",\r\n \"activityId\": \"56e0ac7d-4433-4a86-9cfe-6905af57c05a\",\r\n \"subActivityId\": \"ad63d932-4f9e-4483-9ab8-8ec613cf4164\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"6ed21c2c-224d-44ac-9721-302de763024e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;defe9e59-a9e0-410e-9251-a03a8a2d2f6b ARID=56e0ac7d-4433-4a86-9cfe-6905af57c05a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/4ddc6f31-9ca8-4fa6-acc9-e2c28ed3c59d \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/4ddc6f31-9ca8-4fa6-acc9-e2c28ed3c59d \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/6ed21c2c-224d-44ac-9721-302de763024e \u003e DN.FE.ExecuteQuery/ad63d932-4f9e-4483-9ab8-8ec613cf4164)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[23:38:34.725]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:38:31.5847811Z);\nlet _endTime = datetime(2026-08-24T23:38:31.5847819Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-pgv7kxdwgbs9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:38:34.6523649Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;defe9e59-a9e0-410e-9251-a03a8a2d2f6b\",\r\n \"activityId\": \"56e0ac7d-4433-4a86-9cfe-6905af57c05a\",\r\n \"subActivityId\": \"ad63d932-4f9e-4483-9ab8-8ec613cf4164\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"6ed21c2c-224d-44ac-9721-302de763024e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;defe9e59-a9e0-410e-9251-a03a8a2d2f6b ARID=56e0ac7d-4433-4a86-9cfe-6905af57c05a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/4ddc6f31-9ca8-4fa6-acc9-e2c28ed3c59d \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/4ddc6f31-9ca8-4fa6-acc9-e2c28ed3c59d \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/6ed21c2c-224d-44ac-9721-302de763024e \u003e DN.FE.ExecuteQuery/ad63d932-4f9e-4483-9ab8-8ec613cf4164)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[23:38:51.573]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:38:48.1146150Z);\nlet _endTime = datetime(2026-08-24T23:38:48.1146157Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-pgv7kxdwgbs9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:38:51.5509925Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;54a98755-cda5-4e00-b75a-8a9f4a05e23a\",\r\n \"activityId\": \"07bd916f-8e74-4f33-9544-38554323dccf\",\r\n \"subActivityId\": \"ee30f34f-230c-4bd2-af26-3c459201c20c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"95379035-872f-45ec-8ef7-b6611ac9f3a9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;54a98755-cda5-4e00-b75a-8a9f4a05e23a ARID=07bd916f-8e74-4f33-9544-38554323dccf \u003e GW.Http.CallContext/07bd916f-8e74-4f33-9544-38554323dccf \u003e GW.ExecuteQuery/95379035-872f-45ec-8ef7-b6611ac9f3a9 \u003e DN.FE.ExecuteQuery/ee30f34f-230c-4bd2-af26-3c459201c20c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[23:38:51.682]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:38:48.1146150Z);\nlet _endTime = datetime(2026-08-24T23:38:48.1146157Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-pgv7kxdwgbs9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:38:51.5509925Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;54a98755-cda5-4e00-b75a-8a9f4a05e23a\",\r\n \"activityId\": \"07bd916f-8e74-4f33-9544-38554323dccf\",\r\n \"subActivityId\": \"ee30f34f-230c-4bd2-af26-3c459201c20c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"95379035-872f-45ec-8ef7-b6611ac9f3a9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;54a98755-cda5-4e00-b75a-8a9f4a05e23a ARID=07bd916f-8e74-4f33-9544-38554323dccf \u003e GW.Http.CallContext/07bd916f-8e74-4f33-9544-38554323dccf \u003e GW.ExecuteQuery/95379035-872f-45ec-8ef7-b6611ac9f3a9 \u003e DN.FE.ExecuteQuery/ee30f34f-230c-4bd2-af26-3c459201c20c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ...
22:22:53	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 87c9d3c	e2e	failed	★#6662 (open)	1	e2e (1) query-basic: command failed: exit status 1 22:22:53 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.002s. Expected success, but got an error: <errors.errorString \| 0xc001f02a40>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[23:17:20.322]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:17:17.0513987Z);\nlet _endTime = datetime(2026-08-24T23:17:17.0513993Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-ss9mztkxcbtx'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:17:20.3014069Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;bdf5bb75-f3a4-4922-b252-582f3654b42a\",\r\n \"activityId\": \"5d98f621-5732-4f3e-99d2-21c3ecc8195a\",\r\n \"subActivityId\": \"7a976fd8-5470-4ca6-9a6a-3e041417cc3c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"bb8e1bae-a54b-4f1b-aa1a-86b94820a3f9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;bdf5bb75-f3a4-4922-b252-582f3654b42a ARID=5d98f621-5732-4f3e-99d2-21c3ecc8195a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/2fb79466-7c27-42f3-a6c2-866bddfdc399 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/2fb79466-7c27-42f3-a6c2-866bddfdc399 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/bb8e1bae-a54b-4f1b-aa1a-86b94820a3f9 \u003e DN.FE.ExecuteQuery/7a976fd8-5470-4ca6-9a6a-3e041417cc3c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[23:17:20.389]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:17:17.0513987Z);\nlet _endTime = datetime(2026-08-24T23:17:17.0513993Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-ss9mztkxcbtx'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:17:20.3014069Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;bdf5bb75-f3a4-4922-b252-582f3654b42a\",\r\n \"activityId\": \"5d98f621-5732-4f3e-99d2-21c3ecc8195a\",\r\n \"subActivityId\": \"7a976fd8-5470-4ca6-9a6a-3e041417cc3c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"bb8e1bae-a54b-4f1b-aa1a-86b94820a3f9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;bdf5bb75-f3a4-4922-b252-582f3654b42a ARID=5d98f621-5732-4f3e-99d2-21c3ecc8195a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/2fb79466-7c27-42f3-a6c2-866bddfdc399 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/2fb79466-7c27-42f3-a6c2-866bddfdc399 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/bb8e1bae-a54b-4f1b-aa1a-86b94820a3f9 \u003e DN.FE.ExecuteQuery/7a976fd8-5470-4ca6-9a6a-3e041417cc3c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[23:17:23.477]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:17:20.4650185Z);\nlet _endTime = datetime(2026-08-24T23:17:20.4650189Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-ss9mztkxcbtx'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:17:23.4523754Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;2e051b49-6e32-48db-a742-06c75efc4f02\",\r\n \"activityId\": \"dfb8edfc-26ab-4141-b358-423d3a40e76b\",\r\n \"subActivityId\": \"a58cffec-2cfe-4210-bf95-cf8a49e5d515\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"d7e2f919-501a-4524-8f3e-7eadaba81ac8\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;2e051b49-6e32-48db-a742-06c75efc4f02 ARID=dfb8edfc-26ab-4141-b358-423d3a40e76b \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/7364b8c6-dd3b-462e-a8cd-fdd5d2ebf551 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/7364b8c6-dd3b-462e-a8cd-fdd5d2ebf551 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/d7e2f919-501a-4524-8f3e-7eadaba81ac8 \u003e DN.FE.ExecuteQuery/a58cffec-2cfe-4210-bf95-cf8a49e5d515)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[23:17:23.548]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:17:20.4650185Z);\nlet _endTime = datetime(2026-08-24T23:17:20.4650189Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-ss9mztkxcbtx'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:17:23.4523754Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;2e051b49-6e32-48db-a742-06c75efc4f02\",\r\n \"activityId\": \"dfb8edfc-26ab-4141-b358-423d3a40e76b\",\r\n \"subActivityId\": \"a58cffec-2cfe-4210-bf95-cf8a49e5d515\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"d7e2f919-501a-4524-8f3e-7eadaba81ac8\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;2e051b49-6e32-48db-a742-06c75efc4f02 ARID=dfb8edfc-26ab-4141-b358-423d3a40e76b \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/7364b8c6-dd3b-462e-a8cd-fdd5d2ebf551 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/7364b8c6-dd3b-462e-a8cd-fdd5d2ebf551 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/d7e2f919-501a-4524-8f3e-7eadaba81ac8 \u003e DN.FE.ExecuteQuery/a58cffec-2cfe-4210-bf95-cf8a49e5d515)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[23:17:30.062]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:17:26.7994218Z);\nlet _endTime = datetime(2026-08-24T23:17:26.7994224Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-ss9mztkxcbtx'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:17:30.0404281Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a928551c-c326-4e69-837c-028a41ddae49\",\r\n \"activityId\": \"8a5d47f3-bd6e-4004-afb3-f9fe45c95ae5\",\r\n \"subActivityId\": \"7711a873-92d6-4318-afed-c5650fa8c291\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"19e8c6db-27ba-4efc-ab96-cbc1459b225d\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a928551c-c326-4e69-837c-028a41ddae49 ARID=8a5d47f3-bd6e-4004-afb3-f9fe45c95ae5 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/8a7841e7-91c1-4a08-81b4-539f76ffd5b7 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/8a7841e7-91c1-4a08-81b4-539f76ffd5b7 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/19e8c6db-27ba-4efc-ab96-cbc1459b225d \u003e DN.FE.ExecuteQuery/7711a873-92d6-4318-afed-c5650fa8c291)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[23:17:30.188]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:17:26.7994218Z);\nlet _endTime = datetime(2026-08-24T23:17:26.7994224Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-ss9mztkxcbtx'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:17:30.0404281Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a928551c-c326-4e69-837c-028a41ddae49\",\r\n \"activityId\": \"8a5d47f3-bd6e-4004-afb3-f9fe45c95ae5\",\r\n \"subActivityId\": \"7711a873-92d6-4318-afed-c5650fa8c291\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"19e8c6db-27ba-4efc-ab96-cbc1459b225d\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a928551c-c326-4e69-837c-028a41ddae49 ARID=8a5d47f3-bd6e-4004-afb3-f9fe45c95ae5 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/8a7841e7-91c1-4a08-81b4-539f76ffd5b7 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/8a7841e7-91c1-4a08-81b4-539f76ffd5b7 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/19e8c6db-27ba-4efc-ab96-cbc1459b225d \u003e DN.FE.ExecuteQuery/7711a873-92d6-4318-afed-c5650fa8c291)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.002s. Expected success, but got an error: <errors.errorString \| 0xc001f02a40>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[23:17:20.322]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:17:17.0513987Z);\nlet _endTime = datetime(2026-08-24T23:17:17.0513993Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-ss9mztkxcbtx'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:17:20.3014069Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;bdf5bb75-f3a4-4922-b252-582f3654b42a\",\r\n \"activityId\": \"5d98f621-5732-4f3e-99d2-21c3ecc8195a\",\r\n \"subActivityId\": \"7a976fd8-5470-4ca6-9a6a-3e041417cc3c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"bb8e1bae-a54b-4f1b-aa1a-86b94820a3f9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;bdf5bb75-f3a4-4922-b252-582f3654b42a ARID=5d98f621-5732-4f3e-99d2-21c3ecc8195a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/2fb79466-7c27-42f3-a6c2-866bddfdc399 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/2fb79466-7c27-42f3-a6c2-866bddfdc399 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/bb8e1bae-a54b-4f1b-aa1a-86b94820a3f9 \u003e DN.FE.ExecuteQuery/7a976fd8-5470-4ca6-9a6a-3e041417cc3c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[23:17:20.389]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:17:17.0513987Z);\nlet _endTime = datetime(2026-08-24T23:17:17.0513993Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-ss9mztkxcbtx'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:17:20.3014069Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;bdf5bb75-f3a4-4922-b252-582f3654b42a\",\r\n \"activityId\": \"5d98f621-5732-4f3e-99d2-21c3ecc8195a\",\r\n \"subActivityId\": \"7a976fd8-5470-4ca6-9a6a-3e041417cc3c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"bb8e1bae-a54b-4f1b-aa1a-86b94820a3f9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;bdf5bb75-f3a4-4922-b252-582f3654b42a ARID=5d98f621-5732-4f3e-99d2-21c3ecc8195a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/2fb79466-7c27-42f3-a6c2-866bddfdc399 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/2fb79466-7c27-42f3-a6c2-866bddfdc399 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/bb8e1bae-a54b-4f1b-aa1a-86b94820a3f9 \u003e DN.FE.ExecuteQuery/7a976fd8-5470-4ca6-9a6a-3e041417cc3c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[23:17:23.477]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:17:20.4650185Z);\nlet _endTime = datetime(2026-08-24T23:17:20.4650189Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-ss9mztkxcbtx'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:17:23.4523754Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;2e051b49-6e32-48db-a742-06c75efc4f02\",\r\n \"activityId\": \"dfb8edfc-26ab-4141-b358-423d3a40e76b\",\r\n \"subActivityId\": \"a58cffec-2cfe-4210-bf95-cf8a49e5d515\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"d7e2f919-501a-4524-8f3e-7eadaba81ac8\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;2e051b49-6e32-48db-a742-06c75efc4f02 ARID=dfb8edfc-26ab-4141-b358-423d3a40e76b \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/7364b8c6-dd3b-462e-a8cd-fdd5d2ebf551 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/7364b8c6-dd3b-462e-a8cd-fdd5d2ebf551 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/d7e2f919-501a-4524-8f3e-7eadaba81ac8 \u003e DN.FE.ExecuteQuery/a58cffec-2cfe-4210-bf95-cf8a49e5d515)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[23:17:23.548]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:17:20.4650185Z);\nlet _endTime = datetime(2026-08-24T23:17:20.4650189Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-ss9mztkxcbtx'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:17:23.4523754Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;2e051b49-6e32-48db-a742-06c75efc4f02\",\r\n \"activityId\": \"dfb8edfc-26ab-4141-b358-423d3a40e76b\",\r\n \"subActivityId\": \"a58cffec-2cfe-4210-bf95-cf8a49e5d515\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"d7e2f919-501a-4524-8f3e-7eadaba81ac8\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;2e051b49-6e32-48db-a742-06c75efc4f02 ARID=dfb8edfc-26ab-4141-b358-423d3a40e76b \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/7364b8c6-dd3b-462e-a8cd-fdd5d2ebf551 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/7364b8c6-dd3b-462e-a8cd-fdd5d2ebf551 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/d7e2f919-501a-4524-8f3e-7eadaba81ac8 \u003e DN.FE.ExecuteQuery/a58cffec-2cfe-4210-bf95-cf8a49e5d515)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[23:17:30.062]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:17:26.7994218Z);\nlet _endTime = datetime(2026-08-24T23:17:26.7994224Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-ss9mztkxcbtx'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:17:30.0404281Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a928551c-c326-4e69-837c-028a41ddae49\",\r\n \"activityId\": \"8a5d47f3-bd6e-4004-afb3-f9fe45c95ae5\",\r\n \"subActivityId\": \"7711a873-92d6-4318-afed-c5650fa8c291\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"19e8c6db-27ba-4efc-ab96-cbc1459b225d\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a928551c-c326-4e69-837c-028a41ddae49 ARID=8a5d47f3-bd6e-4004-afb3-f9fe45c95ae5 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/8a7841e7-91c1-4a08-81b4-539f76ffd5b7 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/8a7841e7-91c1-4a08-81b4-539f76ffd5b7 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/19e8c6db-27ba-4efc-ab96-cbc1459b225d \u003e DN.FE.ExecuteQuery/7711a873-92d6-4318-afed-c5650fa8c291)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[23:17:30.188]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T23:17:26.7994218Z);\nlet _endTime = datetime(2026-08-24T23:17:26.7994224Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-ss9mztkxcbtx'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T23:17:30.0404281Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a928551c-c326-4e69-837c-028a41ddae49\",\r\n \"activityId\": \"8a5d47f3-bd6e-4004-afb3-f9fe45c95ae5\",\r\n \"subActivityId\": \"7711a873-92d6-4318-afed-c5650fa8c291\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"19e8c6db-27ba-4efc-ab96-cbc1459b225d\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a928551c-c326-4e69-837c-028a41ddae49 ARID=8a5d47f3-bd6e-4004-afb3-f9fe45c95ae5 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/8a7841e7-91c1-4a08-81b4-539f76ffd5b7 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/8a7841e7-91c1-4a08-81b4-539f76ffd5b7 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/19e8c6db-27ba-4efc-ab96-cbc1459b225d \u003e DN.FE.ExecuteQuery/7711a873-92d6-4318-afed-c5650fa8c291)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ...
21:38:47	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 5fa9bda	e2e	failed	★#6088 (open)	1	e2e (1) query-basic: command failed: exit status 1 21:38:47 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.001s. Expected success, but got an error: <errors.errorString \| 0xc000d24a80>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[22:49:59.230]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:49:56.2562204Z);\nlet _endTime = datetime(2026-08-24T22:49:56.2562208Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-gqhhcj25cdch'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:49:59.2093178Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;c3bef575-02f0-4e36-bf01-e442f3b91c12\",\r\n \"activityId\": \"6ed899e3-526b-485a-86d9-7f14c7afb204\",\r\n \"subActivityId\": \"9aef40a3-6517-4779-af80-2a394ccc34f0\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"0310a9c3-e142-486e-8e89-ea1972124e7e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;c3bef575-02f0-4e36-bf01-e442f3b91c12 ARID=6ed899e3-526b-485a-86d9-7f14c7afb204 \u003e GW.Http.CallContext/6ed899e3-526b-485a-86d9-7f14c7afb204 \u003e GW.ExecuteQuery/0310a9c3-e142-486e-8e89-ea1972124e7e \u003e DN.FE.ExecuteQuery/9aef40a3-6517-4779-af80-2a394ccc34f0)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[22:49:59.276]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:49:56.2562204Z);\nlet _endTime = datetime(2026-08-24T22:49:56.2562208Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-gqhhcj25cdch'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:49:59.2093178Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;c3bef575-02f0-4e36-bf01-e442f3b91c12\",\r\n \"activityId\": \"6ed899e3-526b-485a-86d9-7f14c7afb204\",\r\n \"subActivityId\": \"9aef40a3-6517-4779-af80-2a394ccc34f0\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"0310a9c3-e142-486e-8e89-ea1972124e7e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;c3bef575-02f0-4e36-bf01-e442f3b91c12 ARID=6ed899e3-526b-485a-86d9-7f14c7afb204 \u003e GW.Http.CallContext/6ed899e3-526b-485a-86d9-7f14c7afb204 \u003e GW.ExecuteQuery/0310a9c3-e142-486e-8e89-ea1972124e7e \u003e DN.FE.ExecuteQuery/9aef40a3-6517-4779-af80-2a394ccc34f0)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[22:50:02.572]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:49:59.3623198Z);\nlet _endTime = datetime(2026-08-24T22:49:59.3623202Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-gqhhcj25cdch'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:50:02.5524012Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;338abe06-3616-4676-bb86-93200535d787\",\r\n \"activityId\": \"d9ff4261-86d5-403c-864a-c18c8e8cdd4b\",\r\n \"subActivityId\": \"f710bc94-d353-4e91-a20d-461bdfd636c9\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"60453caa-4266-46ca-b70c-374022fcf35e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;338abe06-3616-4676-bb86-93200535d787 ARID=d9ff4261-86d5-403c-864a-c18c8e8cdd4b \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/29e8a84d-1cb3-4fe7-ab1a-0e3f4cdbb700 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/29e8a84d-1cb3-4fe7-ab1a-0e3f4cdbb700 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/60453caa-4266-46ca-b70c-374022fcf35e \u003e DN.FE.ExecuteQuery/f710bc94-d353-4e91-a20d-461bdfd636c9)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[22:50:02.623]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:49:59.3623198Z);\nlet _endTime = datetime(2026-08-24T22:49:59.3623202Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-gqhhcj25cdch'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:50:02.5524012Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;338abe06-3616-4676-bb86-93200535d787\",\r\n \"activityId\": \"d9ff4261-86d5-403c-864a-c18c8e8cdd4b\",\r\n \"subActivityId\": \"f710bc94-d353-4e91-a20d-461bdfd636c9\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"60453caa-4266-46ca-b70c-374022fcf35e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;338abe06-3616-4676-bb86-93200535d787 ARID=d9ff4261-86d5-403c-864a-c18c8e8cdd4b \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/29e8a84d-1cb3-4fe7-ab1a-0e3f4cdbb700 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/29e8a84d-1cb3-4fe7-ab1a-0e3f4cdbb700 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/60453caa-4266-46ca-b70c-374022fcf35e \u003e DN.FE.ExecuteQuery/f710bc94-d353-4e91-a20d-461bdfd636c9)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[22:50:09.721]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:50:06.2998581Z);\nlet _endTime = datetime(2026-08-24T22:50:06.2998587Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-gqhhcj25cdch'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:50:09.7003481Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;ec1905a7-7ca7-43e4-9299-c9fb7166dc78\",\r\n \"activityId\": \"2c4c24ce-6ca1-4092-b20f-62aa7ce11dac\",\r\n \"subActivityId\": \"828b6fa7-bf28-4ff4-a26e-c89a97292ff3\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"ca11ea7c-ce7e-4449-bfe4-feb2c1797062\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;ec1905a7-7ca7-43e4-9299-c9fb7166dc78 ARID=2c4c24ce-6ca1-4092-b20f-62aa7ce11dac \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/57e05fdd-42df-49c9-9b68-aca74f530e51 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/57e05fdd-42df-49c9-9b68-aca74f530e51 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/ca11ea7c-ce7e-4449-bfe4-feb2c1797062 \u003e DN.FE.ExecuteQuery/828b6fa7-bf28-4ff4-a26e-c89a97292ff3)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[22:50:09.823]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:50:06.2998581Z);\nlet _endTime = datetime(2026-08-24T22:50:06.2998587Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-gqhhcj25cdch'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:50:09.7003481Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;ec1905a7-7ca7-43e4-9299-c9fb7166dc78\",\r\n \"activityId\": \"2c4c24ce-6ca1-4092-b20f-62aa7ce11dac\",\r\n \"subActivityId\": \"828b6fa7-bf28-4ff4-a26e-c89a97292ff3\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"ca11ea7c-ce7e-4449-bfe4-feb2c1797062\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;ec1905a7-7ca7-43e4-9299-c9fb7166dc78 ARID=2c4c24ce-6ca1-4092-b20f-62aa7ce11dac \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/57e05fdd-42df-49c9-9b68-aca74f530e51 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/57e05fdd-42df-49c9-9b68-aca74f530e51 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/ca11ea7c-ce7e-4449-bfe4-feb2c1797062 \u003e DN.FE.ExecuteQuery/828b6fa7-bf28-4ff4-a26e-c89a97292ff3)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.001s. Expected success, but got an error: <errors.errorString \| 0xc000d24a80>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[22:49:59.230]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:49:56.2562204Z);\nlet _endTime = datetime(2026-08-24T22:49:56.2562208Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-gqhhcj25cdch'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:49:59.2093178Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;c3bef575-02f0-4e36-bf01-e442f3b91c12\",\r\n \"activityId\": \"6ed899e3-526b-485a-86d9-7f14c7afb204\",\r\n \"subActivityId\": \"9aef40a3-6517-4779-af80-2a394ccc34f0\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"0310a9c3-e142-486e-8e89-ea1972124e7e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;c3bef575-02f0-4e36-bf01-e442f3b91c12 ARID=6ed899e3-526b-485a-86d9-7f14c7afb204 \u003e GW.Http.CallContext/6ed899e3-526b-485a-86d9-7f14c7afb204 \u003e GW.ExecuteQuery/0310a9c3-e142-486e-8e89-ea1972124e7e \u003e DN.FE.ExecuteQuery/9aef40a3-6517-4779-af80-2a394ccc34f0)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[22:49:59.276]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:49:56.2562204Z);\nlet _endTime = datetime(2026-08-24T22:49:56.2562208Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-gqhhcj25cdch'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:49:59.2093178Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;c3bef575-02f0-4e36-bf01-e442f3b91c12\",\r\n \"activityId\": \"6ed899e3-526b-485a-86d9-7f14c7afb204\",\r\n \"subActivityId\": \"9aef40a3-6517-4779-af80-2a394ccc34f0\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"0310a9c3-e142-486e-8e89-ea1972124e7e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;c3bef575-02f0-4e36-bf01-e442f3b91c12 ARID=6ed899e3-526b-485a-86d9-7f14c7afb204 \u003e GW.Http.CallContext/6ed899e3-526b-485a-86d9-7f14c7afb204 \u003e GW.ExecuteQuery/0310a9c3-e142-486e-8e89-ea1972124e7e \u003e DN.FE.ExecuteQuery/9aef40a3-6517-4779-af80-2a394ccc34f0)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[22:50:02.572]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:49:59.3623198Z);\nlet _endTime = datetime(2026-08-24T22:49:59.3623202Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-gqhhcj25cdch'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:50:02.5524012Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;338abe06-3616-4676-bb86-93200535d787\",\r\n \"activityId\": \"d9ff4261-86d5-403c-864a-c18c8e8cdd4b\",\r\n \"subActivityId\": \"f710bc94-d353-4e91-a20d-461bdfd636c9\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"60453caa-4266-46ca-b70c-374022fcf35e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;338abe06-3616-4676-bb86-93200535d787 ARID=d9ff4261-86d5-403c-864a-c18c8e8cdd4b \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/29e8a84d-1cb3-4fe7-ab1a-0e3f4cdbb700 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/29e8a84d-1cb3-4fe7-ab1a-0e3f4cdbb700 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/60453caa-4266-46ca-b70c-374022fcf35e \u003e DN.FE.ExecuteQuery/f710bc94-d353-4e91-a20d-461bdfd636c9)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[22:50:02.623]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:49:59.3623198Z);\nlet _endTime = datetime(2026-08-24T22:49:59.3623202Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-gqhhcj25cdch'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:50:02.5524012Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;338abe06-3616-4676-bb86-93200535d787\",\r\n \"activityId\": \"d9ff4261-86d5-403c-864a-c18c8e8cdd4b\",\r\n \"subActivityId\": \"f710bc94-d353-4e91-a20d-461bdfd636c9\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"60453caa-4266-46ca-b70c-374022fcf35e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;338abe06-3616-4676-bb86-93200535d787 ARID=d9ff4261-86d5-403c-864a-c18c8e8cdd4b \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/29e8a84d-1cb3-4fe7-ab1a-0e3f4cdbb700 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/29e8a84d-1cb3-4fe7-ab1a-0e3f4cdbb700 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/60453caa-4266-46ca-b70c-374022fcf35e \u003e DN.FE.ExecuteQuery/f710bc94-d353-4e91-a20d-461bdfd636c9)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[22:50:09.721]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:50:06.2998581Z);\nlet _endTime = datetime(2026-08-24T22:50:06.2998587Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-gqhhcj25cdch'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:50:09.7003481Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;ec1905a7-7ca7-43e4-9299-c9fb7166dc78\",\r\n \"activityId\": \"2c4c24ce-6ca1-4092-b20f-62aa7ce11dac\",\r\n \"subActivityId\": \"828b6fa7-bf28-4ff4-a26e-c89a97292ff3\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"ca11ea7c-ce7e-4449-bfe4-feb2c1797062\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;ec1905a7-7ca7-43e4-9299-c9fb7166dc78 ARID=2c4c24ce-6ca1-4092-b20f-62aa7ce11dac \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/57e05fdd-42df-49c9-9b68-aca74f530e51 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/57e05fdd-42df-49c9-9b68-aca74f530e51 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/ca11ea7c-ce7e-4449-bfe4-feb2c1797062 \u003e DN.FE.ExecuteQuery/828b6fa7-bf28-4ff4-a26e-c89a97292ff3)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[22:50:09.823]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:50:06.2998581Z);\nlet _endTime = datetime(2026-08-24T22:50:06.2998587Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-gqhhcj25cdch'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:50:09.7003481Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;ec1905a7-7ca7-43e4-9299-c9fb7166dc78\",\r\n \"activityId\": \"2c4c24ce-6ca1-4092-b20f-62aa7ce11dac\",\r\n \"subActivityId\": \"828b6fa7-bf28-4ff4-a26e-c89a97292ff3\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"ca11ea7c-ce7e-4449-bfe4-feb2c1797062\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;ec1905a7-7ca7-43e4-9299-c9fb7166dc78 ARID=2c4c24ce-6ca1-4092-b20f-62aa7ce11dac \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/57e05fdd-42df-49c9-9b68-aca74f530e51 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/57e05fdd-42df-49c9-9b68-aca74f530e51 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/ca11ea7c-ce7e-4449-bfe4-feb2c1797062 \u003e DN.FE.ExecuteQuery/828b6fa7-bf28-4ff4-a26e-c89a97292ff3)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ...
21:11:15	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 170f768	e2e	failed	★#6128 (open)	1	e2e (1) query-basic: command failed: exit status 1 21:11:15 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 618.943s. Expected success, but got an error: <errors.errorString \| 0xc000aba840>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[22:23:34.777]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:23:31.1340475Z);\nlet _endTime = datetime(2026-08-24T22:23:31.1340479Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lrgztftd846p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:23:34.7544299Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;29db54ce-825b-4bb8-9a70-02b555f68f68\",\r\n \"activityId\": \"64a8c8f1-b43e-49c5-b452-340f5dfc4e2c\",\r\n \"subActivityId\": \"705e5e56-0d99-4165-83da-d9db2f1f98e3\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"c1332563-9620-4ce8-b789-424edc8d3469\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;29db54ce-825b-4bb8-9a70-02b555f68f68 ARID=64a8c8f1-b43e-49c5-b452-340f5dfc4e2c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/fe5bfcde-d8a3-480b-8a59-62f13d3daed7 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/fe5bfcde-d8a3-480b-8a59-62f13d3daed7 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/c1332563-9620-4ce8-b789-424edc8d3469 \u003e DN.FE.ExecuteQuery/705e5e56-0d99-4165-83da-d9db2f1f98e3)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[22:23:34.823]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:23:31.1340475Z);\nlet _endTime = datetime(2026-08-24T22:23:31.1340479Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lrgztftd846p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:23:34.7544299Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;29db54ce-825b-4bb8-9a70-02b555f68f68\",\r\n \"activityId\": \"64a8c8f1-b43e-49c5-b452-340f5dfc4e2c\",\r\n \"subActivityId\": \"705e5e56-0d99-4165-83da-d9db2f1f98e3\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"c1332563-9620-4ce8-b789-424edc8d3469\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;29db54ce-825b-4bb8-9a70-02b555f68f68 ARID=64a8c8f1-b43e-49c5-b452-340f5dfc4e2c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/fe5bfcde-d8a3-480b-8a59-62f13d3daed7 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/fe5bfcde-d8a3-480b-8a59-62f13d3daed7 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/c1332563-9620-4ce8-b789-424edc8d3469 \u003e DN.FE.ExecuteQuery/705e5e56-0d99-4165-83da-d9db2f1f98e3)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[22:23:38.026]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:23:34.8927473Z);\nlet _endTime = datetime(2026-08-24T22:23:34.8927477Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lrgztftd846p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:23:38.0025191Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;025d76c9-ff35-453c-a1d4-d6e1aa8c5ae1\",\r\n \"activityId\": \"ef32b257-f29d-4b0c-81be-6fd6150546f1\",\r\n \"subActivityId\": \"efd6fb31-137b-41e3-b32d-438594e51a18\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"bbf6b235-380f-4084-8005-856fcc5a3fc6\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;025d76c9-ff35-453c-a1d4-d6e1aa8c5ae1 ARID=ef32b257-f29d-4b0c-81be-6fd6150546f1 \u003e GW.Http.CallContext/ef32b257-f29d-4b0c-81be-6fd6150546f1 \u003e GW.ExecuteQuery/bbf6b235-380f-4084-8005-856fcc5a3fc6 \u003e DN.FE.ExecuteQuery/efd6fb31-137b-41e3-b32d-438594e51a18)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[22:23:38.080]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:23:34.8927473Z);\nlet _endTime = datetime(2026-08-24T22:23:34.8927477Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lrgztftd846p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:23:38.0025191Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;025d76c9-ff35-453c-a1d4-d6e1aa8c5ae1\",\r\n \"activityId\": \"ef32b257-f29d-4b0c-81be-6fd6150546f1\",\r\n \"subActivityId\": \"efd6fb31-137b-41e3-b32d-438594e51a18\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"bbf6b235-380f-4084-8005-856fcc5a3fc6\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;025d76c9-ff35-453c-a1d4-d6e1aa8c5ae1 ARID=ef32b257-f29d-4b0c-81be-6fd6150546f1 \u003e GW.Http.CallContext/ef32b257-f29d-4b0c-81be-6fd6150546f1 \u003e GW.ExecuteQuery/bbf6b235-380f-4084-8005-856fcc5a3fc6 \u003e DN.FE.ExecuteQuery/efd6fb31-137b-41e3-b32d-438594e51a18)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[22:23:45.286]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:23:41.6923571Z);\nlet _endTime = datetime(2026-08-24T22:23:41.6923576Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lrgztftd846p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:23:45.2650146Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;2e4b972d-ca6b-440c-8e53-44a9547bd21d\",\r\n \"activityId\": \"68da6022-33f6-4d2a-8f0b-b8f75a25481d\",\r\n \"subActivityId\": \"ab90ba5e-3045-41f0-a9b2-b9bec85664f5\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"e5180dcb-f6ed-4270-b95c-603ec7e8fcd0\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;2e4b972d-ca6b-440c-8e53-44a9547bd21d ARID=68da6022-33f6-4d2a-8f0b-b8f75a25481d \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/e23736d1-fb0e-41f0-9f0e-c1d42a7dabbe \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/e23736d1-fb0e-41f0-9f0e-c1d42a7dabbe \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/e5180dcb-f6ed-4270-b95c-603ec7e8fcd0 \u003e DN.FE.ExecuteQuery/ab90ba5e-3045-41f0-a9b2-b9bec85664f5)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[22:23:45.384]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:23:41.6923571Z);\nlet _endTime = datetime(2026-08-24T22:23:41.6923576Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lrgztftd846p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:23:45.2650146Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;2e4b972d-ca6b-440c-8e53-44a9547bd21d\",\r\n \"activityId\": \"68da6022-33f6-4d2a-8f0b-b8f75a25481d\",\r\n \"subActivityId\": \"ab90ba5e-3045-41f0-a9b2-b9bec85664f5\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"e5180dcb-f6ed-4270-b95c-603ec7e8fcd0\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;2e4b972d-ca6b-440c-8e53-44a9547bd21d ARID=68da6022-33f6-4d2a-8f0b-b8f75a25481d \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/e23736d1-fb0e-41f0-9f0e-c1d42a7dabbe \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/e23736d1-fb0e-41f0-9f0e-c1d42a7dabbe \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/e5180dcb-f6ed-4270-b95c-603ec7e8fcd0 \u003e DN.FE.ExecuteQuery/ab90ba5e-3045-41f0-a9b2-b9bec85664f5)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 618.943s. Expected success, but got an error: <errors.errorString \| 0xc000aba840>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[22:23:34.777]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:23:31.1340475Z);\nlet _endTime = datetime(2026-08-24T22:23:31.1340479Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lrgztftd846p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:23:34.7544299Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;29db54ce-825b-4bb8-9a70-02b555f68f68\",\r\n \"activityId\": \"64a8c8f1-b43e-49c5-b452-340f5dfc4e2c\",\r\n \"subActivityId\": \"705e5e56-0d99-4165-83da-d9db2f1f98e3\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"c1332563-9620-4ce8-b789-424edc8d3469\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;29db54ce-825b-4bb8-9a70-02b555f68f68 ARID=64a8c8f1-b43e-49c5-b452-340f5dfc4e2c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/fe5bfcde-d8a3-480b-8a59-62f13d3daed7 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/fe5bfcde-d8a3-480b-8a59-62f13d3daed7 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/c1332563-9620-4ce8-b789-424edc8d3469 \u003e DN.FE.ExecuteQuery/705e5e56-0d99-4165-83da-d9db2f1f98e3)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[22:23:34.823]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:23:31.1340475Z);\nlet _endTime = datetime(2026-08-24T22:23:31.1340479Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lrgztftd846p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:23:34.7544299Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;29db54ce-825b-4bb8-9a70-02b555f68f68\",\r\n \"activityId\": \"64a8c8f1-b43e-49c5-b452-340f5dfc4e2c\",\r\n \"subActivityId\": \"705e5e56-0d99-4165-83da-d9db2f1f98e3\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"c1332563-9620-4ce8-b789-424edc8d3469\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;29db54ce-825b-4bb8-9a70-02b555f68f68 ARID=64a8c8f1-b43e-49c5-b452-340f5dfc4e2c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/fe5bfcde-d8a3-480b-8a59-62f13d3daed7 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/fe5bfcde-d8a3-480b-8a59-62f13d3daed7 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/c1332563-9620-4ce8-b789-424edc8d3469 \u003e DN.FE.ExecuteQuery/705e5e56-0d99-4165-83da-d9db2f1f98e3)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[22:23:38.026]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:23:34.8927473Z);\nlet _endTime = datetime(2026-08-24T22:23:34.8927477Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lrgztftd846p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:23:38.0025191Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;025d76c9-ff35-453c-a1d4-d6e1aa8c5ae1\",\r\n \"activityId\": \"ef32b257-f29d-4b0c-81be-6fd6150546f1\",\r\n \"subActivityId\": \"efd6fb31-137b-41e3-b32d-438594e51a18\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"bbf6b235-380f-4084-8005-856fcc5a3fc6\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;025d76c9-ff35-453c-a1d4-d6e1aa8c5ae1 ARID=ef32b257-f29d-4b0c-81be-6fd6150546f1 \u003e GW.Http.CallContext/ef32b257-f29d-4b0c-81be-6fd6150546f1 \u003e GW.ExecuteQuery/bbf6b235-380f-4084-8005-856fcc5a3fc6 \u003e DN.FE.ExecuteQuery/efd6fb31-137b-41e3-b32d-438594e51a18)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[22:23:38.080]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:23:34.8927473Z);\nlet _endTime = datetime(2026-08-24T22:23:34.8927477Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lrgztftd846p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:23:38.0025191Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;025d76c9-ff35-453c-a1d4-d6e1aa8c5ae1\",\r\n \"activityId\": \"ef32b257-f29d-4b0c-81be-6fd6150546f1\",\r\n \"subActivityId\": \"efd6fb31-137b-41e3-b32d-438594e51a18\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"bbf6b235-380f-4084-8005-856fcc5a3fc6\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;025d76c9-ff35-453c-a1d4-d6e1aa8c5ae1 ARID=ef32b257-f29d-4b0c-81be-6fd6150546f1 \u003e GW.Http.CallContext/ef32b257-f29d-4b0c-81be-6fd6150546f1 \u003e GW.ExecuteQuery/bbf6b235-380f-4084-8005-856fcc5a3fc6 \u003e DN.FE.ExecuteQuery/efd6fb31-137b-41e3-b32d-438594e51a18)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[22:23:45.286]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:23:41.6923571Z);\nlet _endTime = datetime(2026-08-24T22:23:41.6923576Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lrgztftd846p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:23:45.2650146Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;2e4b972d-ca6b-440c-8e53-44a9547bd21d\",\r\n \"activityId\": \"68da6022-33f6-4d2a-8f0b-b8f75a25481d\",\r\n \"subActivityId\": \"ab90ba5e-3045-41f0-a9b2-b9bec85664f5\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"e5180dcb-f6ed-4270-b95c-603ec7e8fcd0\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;2e4b972d-ca6b-440c-8e53-44a9547bd21d ARID=68da6022-33f6-4d2a-8f0b-b8f75a25481d \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/e23736d1-fb0e-41f0-9f0e-c1d42a7dabbe \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/e23736d1-fb0e-41f0-9f0e-c1d42a7dabbe \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/e5180dcb-f6ed-4270-b95c-603ec7e8fcd0 \u003e DN.FE.ExecuteQuery/ab90ba5e-3045-41f0-a9b2-b9bec85664f5)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[22:23:45.384]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:23:41.6923571Z);\nlet _endTime = datetime(2026-08-24T22:23:41.6923576Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lrgztftd846p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:23:45.2650146Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;2e4b972d-ca6b-440c-8e53-44a9547bd21d\",\r\n \"activityId\": \"68da6022-33f6-4d2a-8f0b-b8f75a25481d\",\r\n \"subActivityId\": \"ab90ba5e-3045-41f0-a9b2-b9bec85664f5\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"e5180dcb-f6ed-4270-b95c-603ec7e8fcd0\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;2e4b972d-ca6b-440c-8e53-44a9547bd21d ARID=68da6022-33f6-4d2a-8f0b-b8f75a25481d \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/e23736d1-fb0e-41f0-9f0e-c1d42a7dabbe \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/e23736d1-fb0e-41f0-9f0e-c1d42a7dabbe \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/e5180dcb-f6ed-4270-b95c-603ec7e8fcd0 \u003e DN.FE.ExecuteQuery/ab90ba5e-3045-41f0-a9b2-b9bec85664f5)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ...
21:05:15	pull-ci-Azure-ARO-HCP-main-e2e-parallel head d5a9f48	alert, e2e	failed	★#6648 (open)	2	e2e (1) query-basic: command failed: exit status 1 21:05:15 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.007s. Expected success, but got an error: <errors.errorString \| 0xc000e70850>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[22:29:34.250]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:29:31.0597247Z);\nlet _endTime = datetime(2026-08-24T22:29:31.0597254Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-g8cd8qls4hh6'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:29:34.2294855Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;ada62b8a-d2fa-4201-bf57-a131a1316d95\",\r\n \"activityId\": \"384e9969-d946-4906-9998-c82ec6a77993\",\r\n \"subActivityId\": \"789ba336-45e5-4f69-84c4-f02612a2f6cc\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f0abbf07-bd6a-4728-acfe-5cef65fcda63\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;ada62b8a-d2fa-4201-bf57-a131a1316d95 ARID=384e9969-d946-4906-9998-c82ec6a77993 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b13ab986-6ab1-4d9f-bb60-653a61d05f70 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b13ab986-6ab1-4d9f-bb60-653a61d05f70 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/f0abbf07-bd6a-4728-acfe-5cef65fcda63 \u003e DN.FE.ExecuteQuery/789ba336-45e5-4f69-84c4-f02612a2f6cc)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[22:29:34.301]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:29:31.0597247Z);\nlet _endTime = datetime(2026-08-24T22:29:31.0597254Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-g8cd8qls4hh6'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:29:34.2294855Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;ada62b8a-d2fa-4201-bf57-a131a1316d95\",\r\n \"activityId\": \"384e9969-d946-4906-9998-c82ec6a77993\",\r\n \"subActivityId\": \"789ba336-45e5-4f69-84c4-f02612a2f6cc\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f0abbf07-bd6a-4728-acfe-5cef65fcda63\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;ada62b8a-d2fa-4201-bf57-a131a1316d95 ARID=384e9969-d946-4906-9998-c82ec6a77993 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b13ab986-6ab1-4d9f-bb60-653a61d05f70 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b13ab986-6ab1-4d9f-bb60-653a61d05f70 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/f0abbf07-bd6a-4728-acfe-5cef65fcda63 \u003e DN.FE.ExecuteQuery/789ba336-45e5-4f69-84c4-f02612a2f6cc)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[22:29:37.454]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:29:34.3746843Z);\nlet _endTime = datetime(2026-08-24T22:29:34.3746850Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-g8cd8qls4hh6'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:29:37.4330362Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;c0841516-1b82-4da0-8610-d8a2b44299a3\",\r\n \"activityId\": \"6ba9ca04-11a9-44fa-b3d6-0a774b4600bc\",\r\n \"subActivityId\": \"53e784a5-c3af-4317-954e-e58a974cee6d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"56500fba-f877-4e8b-802e-b37b3fd70d19\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;c0841516-1b82-4da0-8610-d8a2b44299a3 ARID=6ba9ca04-11a9-44fa-b3d6-0a774b4600bc \u003e GW.Http.CallContext/6ba9ca04-11a9-44fa-b3d6-0a774b4600bc \u003e GW.ExecuteQuery/56500fba-f877-4e8b-802e-b37b3fd70d19 \u003e DN.FE.ExecuteQuery/53e784a5-c3af-4317-954e-e58a974cee6d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[22:29:37.504]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:29:34.3746843Z);\nlet _endTime = datetime(2026-08-24T22:29:34.3746850Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-g8cd8qls4hh6'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:29:37.4330362Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;c0841516-1b82-4da0-8610-d8a2b44299a3\",\r\n \"activityId\": \"6ba9ca04-11a9-44fa-b3d6-0a774b4600bc\",\r\n \"subActivityId\": \"53e784a5-c3af-4317-954e-e58a974cee6d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"56500fba-f877-4e8b-802e-b37b3fd70d19\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;c0841516-1b82-4da0-8610-d8a2b44299a3 ARID=6ba9ca04-11a9-44fa-b3d6-0a774b4600bc \u003e GW.Http.CallContext/6ba9ca04-11a9-44fa-b3d6-0a774b4600bc \u003e GW.ExecuteQuery/56500fba-f877-4e8b-802e-b37b3fd70d19 \u003e DN.FE.ExecuteQuery/53e784a5-c3af-4317-954e-e58a974cee6d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[22:29:44.411]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:29:40.9730143Z);\nlet _endTime = datetime(2026-08-24T22:29:40.9730148Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-g8cd8qls4hh6'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:29:44.3871339Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;6c96089c-3f08-4ae2-84cd-e6920c98e2f4\",\r\n \"activityId\": \"cda408ef-22ce-4bd6-bad1-04a3011f4f86\",\r\n \"subActivityId\": \"49ad97cc-b49e-4c1d-a94b-bdee0e82195a\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"67c19916-b811-4022-8b94-037ea0c81eae\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;6c96089c-3f08-4ae2-84cd-e6920c98e2f4 ARID=cda408ef-22ce-4bd6-bad1-04a3011f4f86 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b6df6c94-d688-4ac3-986f-aeaddd3706e7 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b6df6c94-d688-4ac3-986f-aeaddd3706e7 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/67c19916-b811-4022-8b94-037ea0c81eae \u003e DN.FE.ExecuteQuery/49ad97cc-b49e-4c1d-a94b-bdee0e82195a)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[22:29:44.518]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:29:40.9730143Z);\nlet _endTime = datetime(2026-08-24T22:29:40.9730148Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-g8cd8qls4hh6'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:29:44.3871339Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;6c96089c-3f08-4ae2-84cd-e6920c98e2f4\",\r\n \"activityId\": \"cda408ef-22ce-4bd6-bad1-04a3011f4f86\",\r\n \"subActivityId\": \"49ad97cc-b49e-4c1d-a94b-bdee0e82195a\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"67c19916-b811-4022-8b94-037ea0c81eae\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;6c96089c-3f08-4ae2-84cd-e6920c98e2f4 ARID=cda408ef-22ce-4bd6-bad1-04a3011f4f86 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b6df6c94-d688-4ac3-986f-aeaddd3706e7 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b6df6c94-d688-4ac3-986f-aeaddd3706e7 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/67c19916-b811-4022-8b94-037ea0c81eae \u003e DN.FE.ExecuteQuery/49ad97cc-b49e-4c1d-a94b-bdee0e82195a)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.007s. Expected success, but got an error: <errors.errorString \| 0xc000e70850>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[22:29:34.250]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:29:31.0597247Z);\nlet _endTime = datetime(2026-08-24T22:29:31.0597254Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-g8cd8qls4hh6'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:29:34.2294855Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;ada62b8a-d2fa-4201-bf57-a131a1316d95\",\r\n \"activityId\": \"384e9969-d946-4906-9998-c82ec6a77993\",\r\n \"subActivityId\": \"789ba336-45e5-4f69-84c4-f02612a2f6cc\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f0abbf07-bd6a-4728-acfe-5cef65fcda63\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;ada62b8a-d2fa-4201-bf57-a131a1316d95 ARID=384e9969-d946-4906-9998-c82ec6a77993 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b13ab986-6ab1-4d9f-bb60-653a61d05f70 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b13ab986-6ab1-4d9f-bb60-653a61d05f70 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/f0abbf07-bd6a-4728-acfe-5cef65fcda63 \u003e DN.FE.ExecuteQuery/789ba336-45e5-4f69-84c4-f02612a2f6cc)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[22:29:34.301]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:29:31.0597247Z);\nlet _endTime = datetime(2026-08-24T22:29:31.0597254Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-g8cd8qls4hh6'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:29:34.2294855Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;ada62b8a-d2fa-4201-bf57-a131a1316d95\",\r\n \"activityId\": \"384e9969-d946-4906-9998-c82ec6a77993\",\r\n \"subActivityId\": \"789ba336-45e5-4f69-84c4-f02612a2f6cc\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f0abbf07-bd6a-4728-acfe-5cef65fcda63\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;ada62b8a-d2fa-4201-bf57-a131a1316d95 ARID=384e9969-d946-4906-9998-c82ec6a77993 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b13ab986-6ab1-4d9f-bb60-653a61d05f70 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b13ab986-6ab1-4d9f-bb60-653a61d05f70 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/f0abbf07-bd6a-4728-acfe-5cef65fcda63 \u003e DN.FE.ExecuteQuery/789ba336-45e5-4f69-84c4-f02612a2f6cc)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[22:29:37.454]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:29:34.3746843Z);\nlet _endTime = datetime(2026-08-24T22:29:34.3746850Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-g8cd8qls4hh6'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:29:37.4330362Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;c0841516-1b82-4da0-8610-d8a2b44299a3\",\r\n \"activityId\": \"6ba9ca04-11a9-44fa-b3d6-0a774b4600bc\",\r\n \"subActivityId\": \"53e784a5-c3af-4317-954e-e58a974cee6d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"56500fba-f877-4e8b-802e-b37b3fd70d19\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;c0841516-1b82-4da0-8610-d8a2b44299a3 ARID=6ba9ca04-11a9-44fa-b3d6-0a774b4600bc \u003e GW.Http.CallContext/6ba9ca04-11a9-44fa-b3d6-0a774b4600bc \u003e GW.ExecuteQuery/56500fba-f877-4e8b-802e-b37b3fd70d19 \u003e DN.FE.ExecuteQuery/53e784a5-c3af-4317-954e-e58a974cee6d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[22:29:37.504]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:29:34.3746843Z);\nlet _endTime = datetime(2026-08-24T22:29:34.3746850Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-g8cd8qls4hh6'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:29:37.4330362Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;c0841516-1b82-4da0-8610-d8a2b44299a3\",\r\n \"activityId\": \"6ba9ca04-11a9-44fa-b3d6-0a774b4600bc\",\r\n \"subActivityId\": \"53e784a5-c3af-4317-954e-e58a974cee6d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"56500fba-f877-4e8b-802e-b37b3fd70d19\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;c0841516-1b82-4da0-8610-d8a2b44299a3 ARID=6ba9ca04-11a9-44fa-b3d6-0a774b4600bc \u003e GW.Http.CallContext/6ba9ca04-11a9-44fa-b3d6-0a774b4600bc \u003e GW.ExecuteQuery/56500fba-f877-4e8b-802e-b37b3fd70d19 \u003e DN.FE.ExecuteQuery/53e784a5-c3af-4317-954e-e58a974cee6d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[22:29:44.411]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:29:40.9730143Z);\nlet _endTime = datetime(2026-08-24T22:29:40.9730148Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-g8cd8qls4hh6'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:29:44.3871339Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;6c96089c-3f08-4ae2-84cd-e6920c98e2f4\",\r\n \"activityId\": \"cda408ef-22ce-4bd6-bad1-04a3011f4f86\",\r\n \"subActivityId\": \"49ad97cc-b49e-4c1d-a94b-bdee0e82195a\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"67c19916-b811-4022-8b94-037ea0c81eae\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;6c96089c-3f08-4ae2-84cd-e6920c98e2f4 ARID=cda408ef-22ce-4bd6-bad1-04a3011f4f86 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b6df6c94-d688-4ac3-986f-aeaddd3706e7 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b6df6c94-d688-4ac3-986f-aeaddd3706e7 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/67c19916-b811-4022-8b94-037ea0c81eae \u003e DN.FE.ExecuteQuery/49ad97cc-b49e-4c1d-a94b-bdee0e82195a)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[22:29:44.518]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:29:40.9730143Z);\nlet _endTime = datetime(2026-08-24T22:29:40.9730148Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-g8cd8qls4hh6'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:29:44.3871339Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;6c96089c-3f08-4ae2-84cd-e6920c98e2f4\",\r\n \"activityId\": \"cda408ef-22ce-4bd6-bad1-04a3011f4f86\",\r\n \"subActivityId\": \"49ad97cc-b49e-4c1d-a94b-bdee0e82195a\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"67c19916-b811-4022-8b94-037ea0c81eae\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;6c96089c-3f08-4ae2-84cd-e6920c98e2f4 ARID=cda408ef-22ce-4bd6-bad1-04a3011f4f86 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b6df6c94-d688-4ac3-986f-aeaddd3706e7 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b6df6c94-d688-4ac3-986f-aeaddd3706e7 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/67c19916-b811-4022-8b94-037ea0c81eae \u003e DN.FE.ExecuteQuery/49ad97cc-b49e-4c1d-a94b-bdee0e82195a)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... alert (1) alert [svc] BackendControllerRetryHotLoop fired 21:05:15 UTC · failed at alert · test [aro-hcp-observability] [svc] alert BackendControllerRetryHotLoop does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-24T22:22:12Z Ended: 2026-08-24T22:46:11Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j2642688-svc", component="backend", name="observemanagedresourcegroup", severity="warning" Description: Backend controller workqueue observemanagedresourcegroup has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work.
21:03:33	pull-ci-Azure-ARO-HCP-main-e2e-parallel head a660b6e	alert, e2e	failed	★#6660 (open)	8	e2e (7) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 21:03:33 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc00138e500>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-fddhzl9zg9xz" to finish creating: GET https://rp.j9409536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9e47f8d6-28af-497c-8094-400544739634 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9e47f8d6-28af-497c-8094-400544739634", "name": "9e47f8d6-28af-497c-8094-400544739634", "status": "Failed", "startTime": "2026-08-24T21:51:08.657608962Z", "endTime": "2026-08-24T22:10:16.835899624Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc00138e500>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-fddhzl9zg9xz" to finish creating: GET https://rp.j9409536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9e47f8d6-28af-497c-8094-400544739634 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9e47f8d6-28af-497c-8094-400544739634", "name": "9e47f8d6-28af-497c-8094-400544739634", "status": "Failed", "startTime": "2026-08-24T21:51:08.657608962Z", "endTime": "2026-08-24T22:10:16.835899624Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 21:03:33 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260630preview and verify the ingress is internal · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc000e33220>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-ndmwcdqgk685" to finish creating: GET https://rp.j9409536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a097060e-673b-40de-a82a-aa623235d88d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a097060e-673b-40de-a82a-aa623235d88d", "name": "a097060e-673b-40de-a82a-aa623235d88d", "status": "Failed", "startTime": "2026-08-24T21:51:06.650394368Z", "endTime": "2026-08-24T22:10:06.638990375Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc000e33220>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-ndmwcdqgk685" to finish creating: GET https://rp.j9409536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a097060e-673b-40de-a82a-aa623235d88d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a097060e-673b-40de-a82a-aa623235d88d", "name": "a097060e-673b-40de-a82a-aa623235d88d", "status": "Failed", "startTime": "2026-08-24T21:51:06.650394368Z", "endTime": "2026-08-24T22:10:06.638990375Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 21:03:33 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-bvzdjg8t8hhx/cluster-candidate-5-0-vn2bh6 should provision Unexpected error: <fmt.wrapError \| 0xc000eee460>: failed to create HCP cluster cluster-candidate-5-0-vn2bh6: failed waiting for cluster="cluster-candidate-5-0-vn2bh6" in resourcegroup="rg-candidate-5-0-bvzdjg8t8hhx" to finish creating: GET https://rp.j9409536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7131e198-16ae-4d71-b14c-8fd65e02139b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7131e198-16ae-4d71-b14c-8fd65e02139b", "name": "7131e198-16ae-4d71-b14c-8fd65e02139b", "status": "Failed", "startTime": "2026-08-24T21:51:13.862127909Z", "endTime": "2026-08-24T22:10:17.159098241Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-bvzdjg8t8hhx/cluster-candidate-5-0-vn2bh6 should provision Unexpected error: <fmt.wrapError \| 0xc000eee460>: failed to create HCP cluster cluster-candidate-5-0-vn2bh6: failed waiting for cluster="cluster-candidate-5-0-vn2bh6" in resourcegroup="rg-candidate-5-0-bvzdjg8t8hhx" to finish creating: GET https://rp.j9409536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7131e198-16ae-4d71-b14c-8fd65e02139b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7131e198-16ae-4d71-b14c-8fd65e02139b", "name": "7131e198-16ae-4d71-b14c-8fd65e02139b", "status": "Failed", "startTime": "2026-08-24T21:51:13.862127909Z", "endTime": "2026-08-24T22:10:17.159098241Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 21:03:33 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-22lvs576ptrb/cluster-candidate-4-20-qdcf69 should provision Unexpected error: <fmt.wrapError \| 0xc001190ba0>: failed to create HCP cluster cluster-candidate-4-20-qdcf69: failed waiting for cluster="cluster-candidate-4-20-qdcf69" in resourcegroup="rg-candidate-4-20-22lvs576ptrb" to finish creating: GET https://rp.j9409536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dc1dd563-6c7e-4d20-8fce-99e12ec22770 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dc1dd563-6c7e-4d20-8fce-99e12ec22770", "name": "dc1dd563-6c7e-4d20-8fce-99e12ec22770", "status": "Failed", "startTime": "2026-08-24T21:51:04.562343833Z", "endTime": "2026-08-24T22:10:06.652345811Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-22lvs576ptrb/cluster-candidate-4-20-qdcf69 should provision Unexpected error: <fmt.wrapError \| 0xc001190ba0>: failed to create HCP cluster cluster-candidate-4-20-qdcf69: failed waiting for cluster="cluster-candidate-4-20-qdcf69" in resourcegroup="rg-candidate-4-20-22lvs576ptrb" to finish creating: GET https://rp.j9409536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dc1dd563-6c7e-4d20-8fce-99e12ec22770 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dc1dd563-6c7e-4d20-8fce-99e12ec22770", "name": "dc1dd563-6c7e-4d20-8fce-99e12ec22770", "status": "Failed", "startTime": "2026-08-24T21:51:04.562343833Z", "endTime": "2026-08-24T22:10:06.652345811Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred query-basic: command failed: exit status 1 21:03:33 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 631.434s. Expected success, but got an error: <errors.errorString \| 0xc00059a560>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[22:16:23.288]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:16:18.5189597Z);\nlet _endTime = datetime(2026-08-24T22:16:18.5189602Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-vvg442pqd5g9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:16:23.2616891Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;4edcbff7-3ca0-4aa5-818e-511636b3ef0c\",\r\n \"activityId\": \"678135c6-f4c0-42d7-b677-90ad3afd7cab\",\r\n \"subActivityId\": \"f58a13bb-93db-4a42-90ea-ec66ad81b256\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"fd9a7b52-cd1f-44c5-8ff4-f320c1419c1b\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;4edcbff7-3ca0-4aa5-818e-511636b3ef0c ARID=678135c6-f4c0-42d7-b677-90ad3afd7cab \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/7c7bbaad-c9f1-43c4-b0ac-c7b8807c51dd \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/7c7bbaad-c9f1-43c4-b0ac-c7b8807c51dd \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/fd9a7b52-cd1f-44c5-8ff4-f320c1419c1b \u003e DN.FE.ExecuteQuery/f58a13bb-93db-4a42-90ea-ec66ad81b256)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[22:16:23.343]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:16:18.5189597Z);\nlet _endTime = datetime(2026-08-24T22:16:18.5189602Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-vvg442pqd5g9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:16:23.2616891Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;4edcbff7-3ca0-4aa5-818e-511636b3ef0c\",\r\n \"activityId\": \"678135c6-f4c0-42d7-b677-90ad3afd7cab\",\r\n \"subActivityId\": \"f58a13bb-93db-4a42-90ea-ec66ad81b256\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"fd9a7b52-cd1f-44c5-8ff4-f320c1419c1b\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;4edcbff7-3ca0-4aa5-818e-511636b3ef0c ARID=678135c6-f4c0-42d7-b677-90ad3afd7cab \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/7c7bbaad-c9f1-43c4-b0ac-c7b8807c51dd \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/7c7bbaad-c9f1-43c4-b0ac-c7b8807c51dd \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/fd9a7b52-cd1f-44c5-8ff4-f320c1419c1b \u003e DN.FE.ExecuteQuery/f58a13bb-93db-4a42-90ea-ec66ad81b256)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[22:16:28.739]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:16:23.4130746Z);\nlet _endTime = datetime(2026-08-24T22:16:23.4130750Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-vvg442pqd5g9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:16:28.7111996Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;f5e99620-b1af-486e-8321-bf4f7de99b00\",\r\n \"activityId\": \"a7e14c65-4632-4a3e-86a5-525812d02bc7\",\r\n \"subActivityId\": \"932e42f2-36aa-416c-b097-ffd35a2d0018\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"bd188a3a-ae2f-48fd-abfc-42d2265367c4\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;f5e99620-b1af-486e-8321-bf4f7de99b00 ARID=a7e14c65-4632-4a3e-86a5-525812d02bc7 \u003e GW.Http.CallContext/a7e14c65-4632-4a3e-86a5-525812d02bc7 \u003e GW.ExecuteQuery/bd188a3a-ae2f-48fd-abfc-42d2265367c4 \u003e DN.FE.ExecuteQuery/932e42f2-36aa-416c-b097-ffd35a2d0018)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[22:16:29.456]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:16:23.4130746Z);\nlet _endTime = datetime(2026-08-24T22:16:23.4130750Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-vvg442pqd5g9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:16:28.7111996Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;f5e99620-b1af-486e-8321-bf4f7de99b00\",\r\n \"activityId\": \"a7e14c65-4632-4a3e-86a5-525812d02bc7\",\r\n \"subActivityId\": \"932e42f2-36aa-416c-b097-ffd35a2d0018\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"bd188a3a-ae2f-48fd-abfc-42d2265367c4\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;f5e99620-b1af-486e-8321-bf4f7de99b00 ARID=a7e14c65-4632-4a3e-86a5-525812d02bc7 \u003e GW.Http.CallContext/a7e14c65-4632-4a3e-86a5-525812d02bc7 \u003e GW.ExecuteQuery/bd188a3a-ae2f-48fd-abfc-42d2265367c4 \u003e DN.FE.ExecuteQuery/932e42f2-36aa-416c-b097-ffd35a2d0018)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[22:16:45.979]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:16:38.6416976Z);\nlet _endTime = datetime(2026-08-24T22:16:38.6416981Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-vvg442pqd5g9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:16:45.8160979Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;adf5beb9-c32d-4096-9592-6c86179bcb5b\",\r\n \"activityId\": \"9920c01e-ffb2-406f-af33-8e776a18186b\",\r\n \"subActivityId\": \"23f5646a-c800-4fde-a433-c951f4fb3bb4\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"ce428594-daf1-4119-8de8-dc3f4edd8065\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;adf5beb9-c32d-4096-9592-6c86179bcb5b ARID=9920c01e-ffb2-406f-af33-8e776a18186b \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/39567b3b-31f3-4b67-ad88-7e62bc36be0c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/39567b3b-31f3-4b67-ad88-7e62bc36be0c \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/ce428594-daf1-4119-8de8-dc3f4edd8065 \u003e DN.FE.ExecuteQuery/23f5646a-c800-4fde-a433-c951f4fb3bb4)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[22:16:47.476]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:16:38.6416976Z);\nlet _endTime = datetime(2026-08-24T22:16:38.6416981Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-vvg442pqd5g9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:16:45.8160979Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;adf5beb9-c32d-4096-9592-6c86179bcb5b\",\r\n \"activityId\": \"9920c01e-ffb2-406f-af33-8e776a18186b\",\r\n \"subActivityId\": \"23f5646a-c800-4fde-a433-c951f4fb3bb4\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"ce428594-daf1-4119-8de8-dc3f4edd8065\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;adf5beb9-c32d-4096-9592-6c86179bcb5b ARID=9920c01e-ffb2-406f-af33-8e776a18186b \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/39567b3b-31f3-4b67-ad88-7e62bc36be0c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/39567b3b-31f3-4b67-ad88-7e62bc36be0c \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/ce428594-daf1-4119-8de8-dc3f4edd8065 \u003e DN.FE.ExecuteQuery/23f5646a-c800-4fde-a433-c951f4fb3bb4)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 631.434s. Expected success, but got an error: <errors.errorString \| 0xc00059a560>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[22:16:23.288]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:16:18.5189597Z);\nlet _endTime = datetime(2026-08-24T22:16:18.5189602Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-vvg442pqd5g9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:16:23.2616891Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;4edcbff7-3ca0-4aa5-818e-511636b3ef0c\",\r\n \"activityId\": \"678135c6-f4c0-42d7-b677-90ad3afd7cab\",\r\n \"subActivityId\": \"f58a13bb-93db-4a42-90ea-ec66ad81b256\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"fd9a7b52-cd1f-44c5-8ff4-f320c1419c1b\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;4edcbff7-3ca0-4aa5-818e-511636b3ef0c ARID=678135c6-f4c0-42d7-b677-90ad3afd7cab \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/7c7bbaad-c9f1-43c4-b0ac-c7b8807c51dd \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/7c7bbaad-c9f1-43c4-b0ac-c7b8807c51dd \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/fd9a7b52-cd1f-44c5-8ff4-f320c1419c1b \u003e DN.FE.ExecuteQuery/f58a13bb-93db-4a42-90ea-ec66ad81b256)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[22:16:23.343]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:16:18.5189597Z);\nlet _endTime = datetime(2026-08-24T22:16:18.5189602Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-vvg442pqd5g9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:16:23.2616891Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;4edcbff7-3ca0-4aa5-818e-511636b3ef0c\",\r\n \"activityId\": \"678135c6-f4c0-42d7-b677-90ad3afd7cab\",\r\n \"subActivityId\": \"f58a13bb-93db-4a42-90ea-ec66ad81b256\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"fd9a7b52-cd1f-44c5-8ff4-f320c1419c1b\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;4edcbff7-3ca0-4aa5-818e-511636b3ef0c ARID=678135c6-f4c0-42d7-b677-90ad3afd7cab \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/7c7bbaad-c9f1-43c4-b0ac-c7b8807c51dd \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/7c7bbaad-c9f1-43c4-b0ac-c7b8807c51dd \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/fd9a7b52-cd1f-44c5-8ff4-f320c1419c1b \u003e DN.FE.ExecuteQuery/f58a13bb-93db-4a42-90ea-ec66ad81b256)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[22:16:28.739]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:16:23.4130746Z);\nlet _endTime = datetime(2026-08-24T22:16:23.4130750Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-vvg442pqd5g9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:16:28.7111996Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;f5e99620-b1af-486e-8321-bf4f7de99b00\",\r\n \"activityId\": \"a7e14c65-4632-4a3e-86a5-525812d02bc7\",\r\n \"subActivityId\": \"932e42f2-36aa-416c-b097-ffd35a2d0018\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"bd188a3a-ae2f-48fd-abfc-42d2265367c4\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;f5e99620-b1af-486e-8321-bf4f7de99b00 ARID=a7e14c65-4632-4a3e-86a5-525812d02bc7 \u003e GW.Http.CallContext/a7e14c65-4632-4a3e-86a5-525812d02bc7 \u003e GW.ExecuteQuery/bd188a3a-ae2f-48fd-abfc-42d2265367c4 \u003e DN.FE.ExecuteQuery/932e42f2-36aa-416c-b097-ffd35a2d0018)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[22:16:29.456]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:16:23.4130746Z);\nlet _endTime = datetime(2026-08-24T22:16:23.4130750Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-vvg442pqd5g9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:16:28.7111996Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;f5e99620-b1af-486e-8321-bf4f7de99b00\",\r\n \"activityId\": \"a7e14c65-4632-4a3e-86a5-525812d02bc7\",\r\n \"subActivityId\": \"932e42f2-36aa-416c-b097-ffd35a2d0018\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"bd188a3a-ae2f-48fd-abfc-42d2265367c4\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;f5e99620-b1af-486e-8321-bf4f7de99b00 ARID=a7e14c65-4632-4a3e-86a5-525812d02bc7 \u003e GW.Http.CallContext/a7e14c65-4632-4a3e-86a5-525812d02bc7 \u003e GW.ExecuteQuery/bd188a3a-ae2f-48fd-abfc-42d2265367c4 \u003e DN.FE.ExecuteQuery/932e42f2-36aa-416c-b097-ffd35a2d0018)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[22:16:45.979]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:16:38.6416976Z);\nlet _endTime = datetime(2026-08-24T22:16:38.6416981Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-vvg442pqd5g9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:16:45.8160979Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;adf5beb9-c32d-4096-9592-6c86179bcb5b\",\r\n \"activityId\": \"9920c01e-ffb2-406f-af33-8e776a18186b\",\r\n \"subActivityId\": \"23f5646a-c800-4fde-a433-c951f4fb3bb4\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"ce428594-daf1-4119-8de8-dc3f4edd8065\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;adf5beb9-c32d-4096-9592-6c86179bcb5b ARID=9920c01e-ffb2-406f-af33-8e776a18186b \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/39567b3b-31f3-4b67-ad88-7e62bc36be0c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/39567b3b-31f3-4b67-ad88-7e62bc36be0c \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/ce428594-daf1-4119-8de8-dc3f4edd8065 \u003e DN.FE.ExecuteQuery/23f5646a-c800-4fde-a433-c951f4fb3bb4)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[22:16:47.476]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T22:16:38.6416976Z);\nlet _endTime = datetime(2026-08-24T22:16:38.6416981Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-vvg442pqd5g9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T22:16:45.8160979Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;adf5beb9-c32d-4096-9592-6c86179bcb5b\",\r\n \"activityId\": \"9920c01e-ffb2-406f-af33-8e776a18186b\",\r\n \"subActivityId\": \"23f5646a-c800-4fde-a433-c951f4fb3bb4\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"ce428594-daf1-4119-8de8-dc3f4edd8065\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;adf5beb9-c32d-4096-9592-6c86179bcb5b ARID=9920c01e-ffb2-406f-af33-8e776a18186b \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/39567b3b-31f3-4b67-ad88-7e62bc36be0c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/39567b3b-31f3-4b67-ad88-7e62bc36be0c \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/ce428594-daf1-4119-8de8-dc3f4edd8065 \u003e DN.FE.ExecuteQuery/23f5646a-c800-4fde-a433-c951f4fb3bb4)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 21:03:33 UTC · failed at e2e · test FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc001338b40>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-6bbwstsgczsz" to finish creating: GET https://rp.j9409536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/99d622cc-3702-4280-b31f-0b350e37d154 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/99d622cc-3702-4280-b31f-0b350e37d154", "name": "99d622cc-3702-4280-b31f-0b350e37d154", "status": "Failed", "startTime": "2026-08-24T21:51:03.771394068Z", "endTime": "2026-08-24T22:10:06.611292378Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc001338b40>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-6bbwstsgczsz" to finish creating: GET https://rp.j9409536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/99d622cc-3702-4280-b31f-0b350e37d154 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/99d622cc-3702-4280-b31f-0b350e37d154", "name": "99d622cc-3702-4280-b31f-0b350e37d154", "status": "Failed", "startTime": "2026-08-24T21:51:03.771394068Z", "endTime": "2026-08-24T22:10:06.611292378Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 21:03:33 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--w9hvk7h29vl7/cluster-hs-pre-4vwvb8 should provision Unexpected error: <fmt.wrapError \| 0xc0001b9780>: failed to create HCP cluster cluster-hs-pre-4vwvb8: failed waiting for cluster="cluster-hs-pre-4vwvb8" in resourcegroup="rg-hs-pre--w9hvk7h29vl7" to finish creating: GET https://rp.j9409536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ab5d6fca-4327-4d5b-8822-0b904b70d00b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ab5d6fca-4327-4d5b-8822-0b904b70d00b", "name": "ab5d6fca-4327-4d5b-8822-0b904b70d00b", "status": "Failed", "startTime": "2026-08-24T21:50:43.487651081Z", "endTime": "2026-08-24T22:09:46.584847851Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--w9hvk7h29vl7/cluster-hs-pre-4vwvb8 should provision Unexpected error: <fmt.wrapError \| 0xc0001b9780>: failed to create HCP cluster cluster-hs-pre-4vwvb8: failed waiting for cluster="cluster-hs-pre-4vwvb8" in resourcegroup="rg-hs-pre--w9hvk7h29vl7" to finish creating: GET https://rp.j9409536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ab5d6fca-4327-4d5b-8822-0b904b70d00b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ab5d6fca-4327-4d5b-8822-0b904b70d00b", "name": "ab5d6fca-4327-4d5b-8822-0b904b70d00b", "status": "Failed", "startTime": "2026-08-24T21:50:43.487651081Z", "endTime": "2026-08-24T22:09:46.584847851Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 21:03:33 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-08-24T22:04:58Z Ended: 2026-08-24T22:16:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9409536-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd6tku01d4e5c0s0k2ocfdtf91cpr8l-d3q6r6q7q0m9f7w", pod="router-57846fcf5b-n5mlw", severity="warning" Description: Pod ocm-arohcpci01-2sd6tku01d4e5c0s0k2ocfdtf91cpr8l-d3q6r6q7q0m9f7w/router-57846fcf5b-n5mlw has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-08-24T22:04:58Z Ended: 2026-08-24T22:16:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9409536-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd6td57g5fetk5hnhg543eg0jfi9lrt-j5s2f3c0f5x7m8z", pod="router-7586ffccfd-4mvlg", severity="warning" Description: Pod ocm-arohcpci01-2sd6td57g5fetk5hnhg543eg0jfi9lrt-j5s2f3c0f5x7m8z/router-7586ffccfd-4mvlg has been in a non-ready state for longer than 5 minutes.
20:43:22	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 3ec9ecd	alert, e2e	failed	★#6364 (open)	9	e2e (8) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 20:43:22 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:148]: failed to create HCP cluster np-version-upgrade-cluster-vkwj8z with version 4.21.30 Unexpected error: <fmt.wrapError \| 0xc000bca9c0>: failed to create HCP cluster np-version-upgrade-cluster-vkwj8z: failed waiting for cluster="np-version-upgrade-cluster-vkwj8z" in resourcegroup="rg-np-version-upgrade-vkwj8z-pvn2kjtfhwm4" to finish creating: GET https://rp.j6887040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9ca593c6-172c-442e-bdb4-2b96ae721c04 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9ca593c6-172c-442e-bdb4-2b96ae721c04", "name": "9ca593c6-172c-442e-bdb4-2b96ae721c04", "status": "Failed", "startTime": "2026-08-24T21:29:02.41159338Z", "endTime": "2026-08-24T21:48:06.346460728Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:148]: failed to create HCP cluster np-version-upgrade-cluster-vkwj8z with version 4.21.30 Unexpected error: <fmt.wrapError \| 0xc000bca9c0>: failed to create HCP cluster np-version-upgrade-cluster-vkwj8z: failed waiting for cluster="np-version-upgrade-cluster-vkwj8z" in resourcegroup="rg-np-version-upgrade-vkwj8z-pvn2kjtfhwm4" to finish creating: GET https://rp.j6887040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9ca593c6-172c-442e-bdb4-2b96ae721c04 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9ca593c6-172c-442e-bdb4-2b96ae721c04", "name": "9ca593c6-172c-442e-bdb4-2b96ae721c04", "status": "Failed", "startTime": "2026-08-24T21:29:02.41159338Z", "endTime": "2026-08-24T21:48:06.346460728Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 20:43:22 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:93]: failed to create HCP cluster "sre-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000a80440>: failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-kb45vnq4q6dl" to finish creating: GET https://rp.j6887040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9f3d7388-91da-4d85-961d-9ff931b5b33b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9f3d7388-91da-4d85-961d-9ff931b5b33b", "name": "9f3d7388-91da-4d85-961d-9ff931b5b33b", "status": "Failed", "startTime": "2026-08-24T21:28:50.487332824Z", "endTime": "2026-08-24T21:47:56.254666995Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:93]: failed to create HCP cluster "sre-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000a80440>: failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-kb45vnq4q6dl" to finish creating: GET https://rp.j6887040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9f3d7388-91da-4d85-961d-9ff931b5b33b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9f3d7388-91da-4d85-961d-9ff931b5b33b", "name": "9f3d7388-91da-4d85-961d-9ff931b5b33b", "status": "Failed", "startTime": "2026-08-24T21:28:50.487332824Z", "endTime": "2026-08-24T21:47:56.254666995Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred query-basic: command failed: exit status 1 20:43:22 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:101]: Timed out after 618.080s. Expected success, but got an error: <errors.errorString \| 0xc0022162a0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[21:53:58.338]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:53:53.8591464Z);\nlet _endTime = datetime(2026-08-24T21:53:53.8591470Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-4448xk6286t9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:53:58.3085967Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;ea05e4ed-8aab-4cbf-ac1f-16ee87838ed4\",\r\n \"activityId\": \"31110b47-1689-4099-b0ab-73175d25e75e\",\r\n \"subActivityId\": \"74c66fb0-9a3d-480c-9f12-67f385ace1d6\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"0c8fa664-22c8-4399-81ea-fdd064d89c4c\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;ea05e4ed-8aab-4cbf-ac1f-16ee87838ed4 ARID=31110b47-1689-4099-b0ab-73175d25e75e \u003e GW.Http.CallContext/31110b47-1689-4099-b0ab-73175d25e75e \u003e GW.ExecuteQuery/0c8fa664-22c8-4399-81ea-fdd064d89c4c \u003e DN.FE.ExecuteQuery/74c66fb0-9a3d-480c-9f12-67f385ace1d6)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:53:58.338]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "context canceled", "name": "cosmosResourceSnapshots" }�[0m �[37m[21:53:58.430]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:53:53.8591464Z);\nlet _endTime = datetime(2026-08-24T21:53:53.8591470Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-4448xk6286t9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:53:58.3085967Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;ea05e4ed-8aab-4cbf-ac1f-16ee87838ed4\",\r\n \"activityId\": \"31110b47-1689-4099-b0ab-73175d25e75e\",\r\n \"subActivityId\": \"74c66fb0-9a3d-480c-9f12-67f385ace1d6\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"0c8fa664-22c8-4399-81ea-fdd064d89c4c\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;ea05e4ed-8aab-4cbf-ac1f-16ee87838ed4 ARID=31110b47-1689-4099-b0ab-73175d25e75e \u003e GW.Http.CallContext/31110b47-1689-4099-b0ab-73175d25e75e \u003e GW.ExecuteQuery/0c8fa664-22c8-4399-81ea-fdd064d89c4c \u003e DN.FE.ExecuteQuery/74c66fb0-9a3d-480c-9f12-67f385ace1d6)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[21:54:02.554]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:53:58.6053160Z);\nlet _endTime = datetime(2026-08-24T21:53:58.6053166Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-4448xk6286t9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:54:02.5313557Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;57a9070e-2d96-422b-978f-0aacac128a23\",\r\n \"activityId\": \"6272b7b7-e9aa-464e-8a01-e9e1b3804dec\",\r\n \"subActivityId\": \"2bc2139c-c96a-4588-934c-86452b080a0b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"aac4e537-146b-4268-a5da-1277328712aa\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;57a9070e-2d96-422b-978f-0aacac128a23 ARID=6272b7b7-e9aa-464e-8a01-e9e1b3804dec \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/5f8af6a8-db30-4af7-9126-fbbcbeb149c3 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/5f8af6a8-db30-4af7-9126-fbbcbeb149c3 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/aac4e537-146b-4268-a5da-1277328712aa \u003e DN.FE.ExecuteQuery/2bc2139c-c96a-4588-934c-86452b080a0b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:54:02.554]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "context canceled", "name": "cosmosResourceSnapshots" }�[0m �[37m[21:54:02.628]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:53:58.6053160Z);\nlet _endTime = datetime(2026-08-24T21:53:58.6053166Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-4448xk6286t9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:54:02.5313557Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;57a9070e-2d96-422b-978f-0aacac128a23\",\r\n \"activityId\": \"6272b7b7-e9aa-464e-8a01-e9e1b3804dec\",\r\n \"subActivityId\": \"2bc2139c-c96a-4588-934c-86452b080a0b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"aac4e537-146b-4268-a5da-1277328712aa\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;57a9070e-2d96-422b-978f-0aacac128a23 ARID=6272b7b7-e9aa-464e-8a01-e9e1b3804dec \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/5f8af6a8-db30-4af7-9126-fbbcbeb149c3 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/5f8af6a8-db30-4af7-9126-fbbcbeb149c3 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/aac4e537-146b-4268-a5da-1277328712aa \u003e DN.FE.ExecuteQuery/2bc2139c-c96a-4588-934c-86452b080a0b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[21:54:11.406]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:54:07.4434533Z);\nlet _endTime = datetime(2026-08-24T21:54:07.4434539Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-4448xk6286t9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:54:11.3785034Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;98a932a5-07da-4891-ae17-2c0a43b81d7d\",\r\n \"activityId\": \"de64d66e-7dd9-4de8-b52a-d9924cfcf263\",\r\n \"subActivityId\": \"1fe28bac-d3b5-487e-8112-1edd1a6e2076\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"7bc313c6-07c1-4074-9710-48a250fa6b83\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;98a932a5-07da-4891-ae17-2c0a43b81d7d ARID=de64d66e-7dd9-4de8-b52a-d9924cfcf263 \u003e GW.Http.CallContext/de64d66e-7dd9-4de8-b52a-d9924cfcf263 \u003e GW.ExecuteQuery/7bc313c6-07c1-4074-9710-48a250fa6b83 \u003e DN.FE.ExecuteQuery/1fe28bac-d3b5-487e-8112-1edd1a6e2076)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:54:11.407]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "context canceled", "name": "cosmosResourceSnapshots" }�[0m �[37m[21:54:11.535]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:54:07.4434533Z);\nlet _endTime = datetime(2026-08-24T21:54:07.4434539Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-4448xk6286t9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:54:11.3785034Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;98a932a5-07da-4891-ae17-2c0a43b81d7d\",\r\n \"activityId\": \"de64d66e-7dd9-4de8-b52a-d9924cfcf263\",\r\n \"subActivityId\": \"1fe28bac-d3b5-487e-8112-1edd1a6e2076\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"7bc313c6-07c1-4074-9710-48a250fa6b83\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;98a932a5-07da-4891-ae17-2c0a43b81d7d ARID=de64d66e-7dd9-4de8-b52a-d9924cfcf263 \u003e GW.Http.CallContext/de64d66e-7dd9-4de8-b52a-d9924cfcf263 \u003e GW.ExecuteQuery/7bc313c6-07c1-4074-9710-48a250fa6b83 \u003e DN.FE.ExecuteQuery/1fe28bac-d3b5-487e-8112-1edd1a6e2076)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:101]: Timed out after 618.080s. Expected success, but got an error: <errors.errorString \| 0xc0022162a0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[21:53:58.338]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:53:53.8591464Z);\nlet _endTime = datetime(2026-08-24T21:53:53.8591470Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-4448xk6286t9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:53:58.3085967Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;ea05e4ed-8aab-4cbf-ac1f-16ee87838ed4\",\r\n \"activityId\": \"31110b47-1689-4099-b0ab-73175d25e75e\",\r\n \"subActivityId\": \"74c66fb0-9a3d-480c-9f12-67f385ace1d6\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"0c8fa664-22c8-4399-81ea-fdd064d89c4c\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;ea05e4ed-8aab-4cbf-ac1f-16ee87838ed4 ARID=31110b47-1689-4099-b0ab-73175d25e75e \u003e GW.Http.CallContext/31110b47-1689-4099-b0ab-73175d25e75e \u003e GW.ExecuteQuery/0c8fa664-22c8-4399-81ea-fdd064d89c4c \u003e DN.FE.ExecuteQuery/74c66fb0-9a3d-480c-9f12-67f385ace1d6)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:53:58.338]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "context canceled", "name": "cosmosResourceSnapshots" }�[0m �[37m[21:53:58.430]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:53:53.8591464Z);\nlet _endTime = datetime(2026-08-24T21:53:53.8591470Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-4448xk6286t9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:53:58.3085967Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;ea05e4ed-8aab-4cbf-ac1f-16ee87838ed4\",\r\n \"activityId\": \"31110b47-1689-4099-b0ab-73175d25e75e\",\r\n \"subActivityId\": \"74c66fb0-9a3d-480c-9f12-67f385ace1d6\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"0c8fa664-22c8-4399-81ea-fdd064d89c4c\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;ea05e4ed-8aab-4cbf-ac1f-16ee87838ed4 ARID=31110b47-1689-4099-b0ab-73175d25e75e \u003e GW.Http.CallContext/31110b47-1689-4099-b0ab-73175d25e75e \u003e GW.ExecuteQuery/0c8fa664-22c8-4399-81ea-fdd064d89c4c \u003e DN.FE.ExecuteQuery/74c66fb0-9a3d-480c-9f12-67f385ace1d6)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[21:54:02.554]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:53:58.6053160Z);\nlet _endTime = datetime(2026-08-24T21:53:58.6053166Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-4448xk6286t9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:54:02.5313557Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;57a9070e-2d96-422b-978f-0aacac128a23\",\r\n \"activityId\": \"6272b7b7-e9aa-464e-8a01-e9e1b3804dec\",\r\n \"subActivityId\": \"2bc2139c-c96a-4588-934c-86452b080a0b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"aac4e537-146b-4268-a5da-1277328712aa\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;57a9070e-2d96-422b-978f-0aacac128a23 ARID=6272b7b7-e9aa-464e-8a01-e9e1b3804dec \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/5f8af6a8-db30-4af7-9126-fbbcbeb149c3 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/5f8af6a8-db30-4af7-9126-fbbcbeb149c3 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/aac4e537-146b-4268-a5da-1277328712aa \u003e DN.FE.ExecuteQuery/2bc2139c-c96a-4588-934c-86452b080a0b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:54:02.554]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "context canceled", "name": "cosmosResourceSnapshots" }�[0m �[37m[21:54:02.628]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:53:58.6053160Z);\nlet _endTime = datetime(2026-08-24T21:53:58.6053166Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-4448xk6286t9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:54:02.5313557Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;57a9070e-2d96-422b-978f-0aacac128a23\",\r\n \"activityId\": \"6272b7b7-e9aa-464e-8a01-e9e1b3804dec\",\r\n \"subActivityId\": \"2bc2139c-c96a-4588-934c-86452b080a0b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"aac4e537-146b-4268-a5da-1277328712aa\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;57a9070e-2d96-422b-978f-0aacac128a23 ARID=6272b7b7-e9aa-464e-8a01-e9e1b3804dec \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/5f8af6a8-db30-4af7-9126-fbbcbeb149c3 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/5f8af6a8-db30-4af7-9126-fbbcbeb149c3 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/aac4e537-146b-4268-a5da-1277328712aa \u003e DN.FE.ExecuteQuery/2bc2139c-c96a-4588-934c-86452b080a0b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[21:54:11.406]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:54:07.4434533Z);\nlet _endTime = datetime(2026-08-24T21:54:07.4434539Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-4448xk6286t9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:54:11.3785034Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;98a932a5-07da-4891-ae17-2c0a43b81d7d\",\r\n \"activityId\": \"de64d66e-7dd9-4de8-b52a-d9924cfcf263\",\r\n \"subActivityId\": \"1fe28bac-d3b5-487e-8112-1edd1a6e2076\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"7bc313c6-07c1-4074-9710-48a250fa6b83\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;98a932a5-07da-4891-ae17-2c0a43b81d7d ARID=de64d66e-7dd9-4de8-b52a-d9924cfcf263 \u003e GW.Http.CallContext/de64d66e-7dd9-4de8-b52a-d9924cfcf263 \u003e GW.ExecuteQuery/7bc313c6-07c1-4074-9710-48a250fa6b83 \u003e DN.FE.ExecuteQuery/1fe28bac-d3b5-487e-8112-1edd1a6e2076)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:54:11.407]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "context canceled", "name": "cosmosResourceSnapshots" }�[0m �[37m[21:54:11.535]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:54:07.4434533Z);\nlet _endTime = datetime(2026-08-24T21:54:07.4434539Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-4448xk6286t9'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:54:11.3785034Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;98a932a5-07da-4891-ae17-2c0a43b81d7d\",\r\n \"activityId\": \"de64d66e-7dd9-4de8-b52a-d9924cfcf263\",\r\n \"subActivityId\": \"1fe28bac-d3b5-487e-8112-1edd1a6e2076\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"7bc313c6-07c1-4074-9710-48a250fa6b83\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;98a932a5-07da-4891-ae17-2c0a43b81d7d ARID=de64d66e-7dd9-4de8-b52a-d9924cfcf263 \u003e GW.Http.CallContext/de64d66e-7dd9-4de8-b52a-d9924cfcf263 \u003e GW.ExecuteQuery/7bc313c6-07c1-4074-9710-48a250fa6b83 \u003e DN.FE.ExecuteQuery/1fe28bac-d3b5-487e-8112-1edd1a6e2076)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift 20:43:22 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:109]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling Unexpected error: <fmt.wrapError \| 0xc000da71c0>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-wqnnn79bk4qc" to finish creating: GET https://rp.j6887040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e635655f-c9ac-44e1-8232-3550a4c20673 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e635655f-c9ac-44e1-8232-3550a4c20673", "name": "e635655f-c9ac-44e1-8232-3550a4c20673", "status": "Failed", "startTime": "2026-08-24T21:28:57.457434277Z", "endTime": "2026-08-24T21:48:06.423004565Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:109]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling Unexpected error: <fmt.wrapError \| 0xc000da71c0>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-wqnnn79bk4qc" to finish creating: GET https://rp.j6887040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e635655f-c9ac-44e1-8232-3550a4c20673 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e635655f-c9ac-44e1-8232-3550a4c20673", "name": "e635655f-c9ac-44e1-8232-3550a4c20673", "status": "Failed", "startTime": "2026-08-24T21:28:57.457434277Z", "endTime": "2026-08-24T21:48:06.423004565Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; provider Microsoft.RedHatOpenShift 20:43:22 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:304]: failed to create HCP cluster np-noedge-7pzvlw Unexpected error: <fmt.wrapError \| 0xc000e28760>: failed to create HCP cluster np-noedge-7pzvlw: failed waiting for cluster="np-noedge-7pzvlw" in resourcegroup="rg-np-noedge-7pzvlw-hmhh49x6v5k9" to finish creating: GET https://rp.j6887040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d4851c53-78b5-4c3c-8fb8-61c89ed0b4a3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d4851c53-78b5-4c3c-8fb8-61c89ed0b4a3", "name": "d4851c53-78b5-4c3c-8fb8-61c89ed0b4a3", "status": "Failed", "startTime": "2026-08-24T21:29:01.361816726Z", "endTime": "2026-08-24T21:48:06.132587418Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:304]: failed to create HCP cluster np-noedge-7pzvlw Unexpected error: <fmt.wrapError \| 0xc000e28760>: failed to create HCP cluster np-noedge-7pzvlw: failed waiting for cluster="np-noedge-7pzvlw" in resourcegroup="rg-np-noedge-7pzvlw-hmhh49x6v5k9" to finish creating: GET https://rp.j6887040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d4851c53-78b5-4c3c-8fb8-61c89ed0b4a3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d4851c53-78b5-4c3c-8fb8-61c89ed0b4a3", "name": "d4851c53-78b5-4c3c-8fb8-61c89ed0b4a3", "status": "Failed", "startTime": "2026-08-24T21:29:01.361816726Z", "endTime": "2026-08-24T21:48:06.132587418Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; provider Microsoft.RedHatOpenShift 20:43:22 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:95]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0xc000e95be0>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-7w4qb7bx58wj" to finish creating: GET https://rp.j6887040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7c4e3b7e-7834-4bf7-9d07-9139f539790f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7c4e3b7e-7834-4bf7-9d07-9139f539790f", "name": "7c4e3b7e-7834-4bf7-9d07-9139f539790f", "status": "Failed", "startTime": "2026-08-24T21:28:50.851778461Z", "endTime": "2026-08-24T21:47:56.22619205Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:95]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0xc000e95be0>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-7w4qb7bx58wj" to finish creating: GET https://rp.j6887040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7c4e3b7e-7834-4bf7-9d07-9139f539790f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7c4e3b7e-7834-4bf7-9d07-9139f539790f", "name": "7c4e3b7e-7834-4bf7-9d07-9139f539790f", "status": "Failed", "startTime": "2026-08-24T21:28:50.851778461Z", "endTime": "2026-08-24T21:47:56.22619205Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 20:43:22 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-zdwzppg9xnjr/cluster-candidate-4-21-qkclqb should provision Unexpected error: <fmt.wrapError \| 0xc0004e4e40>: failed to create HCP cluster cluster-candidate-4-21-qkclqb: failed waiting for cluster="cluster-candidate-4-21-qkclqb" in resourcegroup="rg-candidate-4-21-zdwzppg9xnjr" to finish creating: GET https://rp.j6887040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2c316ff4-cfc3-45ef-9cbe-7261535c25c4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2c316ff4-cfc3-45ef-9cbe-7261535c25c4", "name": "2c316ff4-cfc3-45ef-9cbe-7261535c25c4", "status": "Failed", "startTime": "2026-08-24T21:29:10.414327144Z", "endTime": "2026-08-24T21:48:16.34399536Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-zdwzppg9xnjr/cluster-candidate-4-21-qkclqb should provision Unexpected error: <fmt.wrapError \| 0xc0004e4e40>: failed to create HCP cluster cluster-candidate-4-21-qkclqb: failed waiting for cluster="cluster-candidate-4-21-qkclqb" in resourcegroup="rg-candidate-4-21-zdwzppg9xnjr" to finish creating: GET https://rp.j6887040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2c316ff4-cfc3-45ef-9cbe-7261535c25c4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2c316ff4-cfc3-45ef-9cbe-7261535c25c4", "name": "2c316ff4-cfc3-45ef-9cbe-7261535c25c4", "status": "Failed", "startTime": "2026-08-24T21:29:10.414327144Z", "endTime": "2026-08-24T21:48:16.34399536Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 20:43:22 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:148]: failed to create HCP cluster np-version-upgrade-cluster-cmkrj8 with version 4.21.30 Unexpected error: <fmt.wrapError \| 0xc001254160>: failed to create HCP cluster np-version-upgrade-cluster-cmkrj8: failed waiting for cluster="np-version-upgrade-cluster-cmkrj8" in resourcegroup="rg-np-version-upgrade-cmkrj8-rmmz8l24rr9j" to finish creating: GET https://rp.j6887040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b6cb9847-c419-4682-a0ae-0563ed4844b9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b6cb9847-c419-4682-a0ae-0563ed4844b9", "name": "b6cb9847-c419-4682-a0ae-0563ed4844b9", "status": "Failed", "startTime": "2026-08-24T21:28:41.610970429Z", "endTime": "2026-08-24T21:47:46.226678817Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:148]: failed to create HCP cluster np-version-upgrade-cluster-cmkrj8 with version 4.21.30 Unexpected error: <fmt.wrapError \| 0xc001254160>: failed to create HCP cluster np-version-upgrade-cluster-cmkrj8: failed waiting for cluster="np-version-upgrade-cluster-cmkrj8" in resourcegroup="rg-np-version-upgrade-cmkrj8-rmmz8l24rr9j" to finish creating: GET https://rp.j6887040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b6cb9847-c419-4682-a0ae-0563ed4844b9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b6cb9847-c419-4682-a0ae-0563ed4844b9", "name": "b6cb9847-c419-4682-a0ae-0563ed4844b9", "status": "Failed", "startTime": "2026-08-24T21:28:41.610970429Z", "endTime": "2026-08-24T21:47:46.226678817Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 20:43:22 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 4 time(s) Firing 1: State: Resolved Started: 2026-08-24T21:39:20Z Ended: 2026-08-24T21:56:20Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j6887040-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd6j294ump2ii7ijvchj6q14avo9t6j-s5a2x8g9m2u0n3v", pod="router-b598fb5ff-bdw2b", severity="warning" Description: Pod ocm-arohcpci01-2sd6j294ump2ii7ijvchj6q14avo9t6j-s5a2x8g9m2u0n3v/router-b598fb5ff-bdw2b has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-08-24T21:40:21Z Ended: 2026-08-24T21:59:17Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j6887040-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd6j9r400h8i6khifr1m2rhln38oon4-x8h6q0k2k5e2h7u", pod="router-688776fb9c-d8gmt", severity="warning" Description: Pod ocm-arohcpci01-2sd6j9r400h8i6khifr1m2rhln38oon4-x8h6q0k2k5e2h7u/router-688776fb9c-d8gmt has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-08-24T21:42:19Z Ended: 2026-08-24T22:02:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j6887040-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd6j7pdvj3o5q8qa4vm85iu5rt4rphu-k0n5x2x4a4h6q2k", pod="router-69c5ffd8ff-bkcml", severity="warning" Description: Pod ocm-arohcpci01-2sd6j7pdvj3o5q8qa4vm85iu5rt4rphu-k0n5x2x4a4h6q2k/router-69c5ffd8ff-bkcml has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-08-24T21:43:18Z Ended: 2026-08-24T22:02:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j6887040-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd6j4q996t8qenevmgfujnj2gprckvj-sre-hcp-cluster", pod="router-768d6d4584-ph9kl", severity="warning" Description: Pod ocm-arohcpci01-2sd6j4q996t8qenevmgfujnj2gprckvj-sre-hcp-cluster/router-768d6d4584-ph9kl has been in a non-ready state for longer than 5 minutes.
20:28:03	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 515e34b	e2e	failed	★#6583 (open)	2	e2e (2) query-basic: command failed: exit status 1 20:28:03 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 604.553s. Expected success, but got an error: <errors.errorString \| 0xc001b12490>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[21:41:43.018]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:41:39.0159138Z);\nlet _endTime = datetime(2026-08-24T21:41:39.0159143Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-sdm9vzp4vv2s'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:41:42.9938618Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;42b7fe0e-1ca7-4eb3-bd94-70b13c5f50a0\",\r\n \"activityId\": \"428709cb-5faa-460e-9fd0-19b365a38358\",\r\n \"subActivityId\": \"c6511ec0-fe87-45c4-8535-22ba47859a18\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"73e933ce-67eb-4635-9995-9c7ad9fa82a5\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;42b7fe0e-1ca7-4eb3-bd94-70b13c5f50a0 ARID=428709cb-5faa-460e-9fd0-19b365a38358 \u003e GW.Http.CallContext/428709cb-5faa-460e-9fd0-19b365a38358 \u003e GW.ExecuteQuery/73e933ce-67eb-4635-9995-9c7ad9fa82a5 \u003e DN.FE.ExecuteQuery/c6511ec0-fe87-45c4-8535-22ba47859a18)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:41:43.076]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:41:39.0159138Z);\nlet _endTime = datetime(2026-08-24T21:41:39.0159143Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-sdm9vzp4vv2s'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:41:42.9938618Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;42b7fe0e-1ca7-4eb3-bd94-70b13c5f50a0\",\r\n \"activityId\": \"428709cb-5faa-460e-9fd0-19b365a38358\",\r\n \"subActivityId\": \"c6511ec0-fe87-45c4-8535-22ba47859a18\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"73e933ce-67eb-4635-9995-9c7ad9fa82a5\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;42b7fe0e-1ca7-4eb3-bd94-70b13c5f50a0 ARID=428709cb-5faa-460e-9fd0-19b365a38358 \u003e GW.Http.CallContext/428709cb-5faa-460e-9fd0-19b365a38358 \u003e GW.ExecuteQuery/73e933ce-67eb-4635-9995-9c7ad9fa82a5 \u003e DN.FE.ExecuteQuery/c6511ec0-fe87-45c4-8535-22ba47859a18)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[21:41:46.333]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:41:43.1465223Z);\nlet _endTime = datetime(2026-08-24T21:41:43.1465227Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-sdm9vzp4vv2s'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:41:46.3064953Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;16d19bd3-8d48-4dca-aa85-db4c9607c99f\",\r\n \"activityId\": \"c7172e67-5da6-46b0-ac83-c1a5b5be4ded\",\r\n \"subActivityId\": \"a1170dba-d226-4772-89b6-9f45fb03d368\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"d3feb915-cfb4-4185-8525-9185496fce8d\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;16d19bd3-8d48-4dca-aa85-db4c9607c99f ARID=c7172e67-5da6-46b0-ac83-c1a5b5be4ded \u003e GW.Http.CallContext/c7172e67-5da6-46b0-ac83-c1a5b5be4ded \u003e GW.ExecuteQuery/d3feb915-cfb4-4185-8525-9185496fce8d \u003e DN.FE.ExecuteQuery/a1170dba-d226-4772-89b6-9f45fb03d368)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:41:46.391]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:41:43.1465223Z);\nlet _endTime = datetime(2026-08-24T21:41:43.1465227Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-sdm9vzp4vv2s'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:41:46.3064953Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;16d19bd3-8d48-4dca-aa85-db4c9607c99f\",\r\n \"activityId\": \"c7172e67-5da6-46b0-ac83-c1a5b5be4ded\",\r\n \"subActivityId\": \"a1170dba-d226-4772-89b6-9f45fb03d368\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"d3feb915-cfb4-4185-8525-9185496fce8d\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;16d19bd3-8d48-4dca-aa85-db4c9607c99f ARID=c7172e67-5da6-46b0-ac83-c1a5b5be4ded \u003e GW.Http.CallContext/c7172e67-5da6-46b0-ac83-c1a5b5be4ded \u003e GW.ExecuteQuery/d3feb915-cfb4-4185-8525-9185496fce8d \u003e DN.FE.ExecuteQuery/a1170dba-d226-4772-89b6-9f45fb03d368)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[21:41:52.877]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:41:49.6902087Z);\nlet _endTime = datetime(2026-08-24T21:41:49.6902091Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-sdm9vzp4vv2s'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:41:52.8452126Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;7ded6e3a-2d78-4f8a-bb3e-3fe5cff3f43c\",\r\n \"activityId\": \"9f24226b-f7d3-4eab-aea7-69e5e64814d8\",\r\n \"subActivityId\": \"2a442ad9-3953-4a76-a5d2-fa2895704a6b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"7111592a-ec51-4e3b-862a-f171de9c1066\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;7ded6e3a-2d78-4f8a-bb3e-3fe5cff3f43c ARID=9f24226b-f7d3-4eab-aea7-69e5e64814d8 \u003e GW.Http.CallContext/9f24226b-f7d3-4eab-aea7-69e5e64814d8 \u003e GW.ExecuteQuery/7111592a-ec51-4e3b-862a-f171de9c1066 \u003e DN.FE.ExecuteQuery/2a442ad9-3953-4a76-a5d2-fa2895704a6b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:41:53.034]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:41:49.6902087Z);\nlet _endTime = datetime(2026-08-24T21:41:49.6902091Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-sdm9vzp4vv2s'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:41:52.8452126Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;7ded6e3a-2d78-4f8a-bb3e-3fe5cff3f43c\",\r\n \"activityId\": \"9f24226b-f7d3-4eab-aea7-69e5e64814d8\",\r\n \"subActivityId\": \"2a442ad9-3953-4a76-a5d2-fa2895704a6b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"7111592a-ec51-4e3b-862a-f171de9c1066\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;7ded6e3a-2d78-4f8a-bb3e-3fe5cff3f43c ARID=9f24226b-f7d3-4eab-aea7-69e5e64814d8 \u003e GW.Http.CallContext/9f24226b-f7d3-4eab-aea7-69e5e64814d8 \u003e GW.ExecuteQuery/7111592a-ec51-4e3b-862a-f171de9c1066 \u003e DN.FE.ExecuteQuery/2a442ad9-3953-4a76-a5d2-fa2895704a6b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 604.553s. Expected success, but got an error: <errors.errorString \| 0xc001b12490>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[21:41:43.018]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:41:39.0159138Z);\nlet _endTime = datetime(2026-08-24T21:41:39.0159143Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-sdm9vzp4vv2s'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:41:42.9938618Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;42b7fe0e-1ca7-4eb3-bd94-70b13c5f50a0\",\r\n \"activityId\": \"428709cb-5faa-460e-9fd0-19b365a38358\",\r\n \"subActivityId\": \"c6511ec0-fe87-45c4-8535-22ba47859a18\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"73e933ce-67eb-4635-9995-9c7ad9fa82a5\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;42b7fe0e-1ca7-4eb3-bd94-70b13c5f50a0 ARID=428709cb-5faa-460e-9fd0-19b365a38358 \u003e GW.Http.CallContext/428709cb-5faa-460e-9fd0-19b365a38358 \u003e GW.ExecuteQuery/73e933ce-67eb-4635-9995-9c7ad9fa82a5 \u003e DN.FE.ExecuteQuery/c6511ec0-fe87-45c4-8535-22ba47859a18)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:41:43.076]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:41:39.0159138Z);\nlet _endTime = datetime(2026-08-24T21:41:39.0159143Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-sdm9vzp4vv2s'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:41:42.9938618Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;42b7fe0e-1ca7-4eb3-bd94-70b13c5f50a0\",\r\n \"activityId\": \"428709cb-5faa-460e-9fd0-19b365a38358\",\r\n \"subActivityId\": \"c6511ec0-fe87-45c4-8535-22ba47859a18\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"73e933ce-67eb-4635-9995-9c7ad9fa82a5\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;42b7fe0e-1ca7-4eb3-bd94-70b13c5f50a0 ARID=428709cb-5faa-460e-9fd0-19b365a38358 \u003e GW.Http.CallContext/428709cb-5faa-460e-9fd0-19b365a38358 \u003e GW.ExecuteQuery/73e933ce-67eb-4635-9995-9c7ad9fa82a5 \u003e DN.FE.ExecuteQuery/c6511ec0-fe87-45c4-8535-22ba47859a18)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[21:41:46.333]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:41:43.1465223Z);\nlet _endTime = datetime(2026-08-24T21:41:43.1465227Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-sdm9vzp4vv2s'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:41:46.3064953Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;16d19bd3-8d48-4dca-aa85-db4c9607c99f\",\r\n \"activityId\": \"c7172e67-5da6-46b0-ac83-c1a5b5be4ded\",\r\n \"subActivityId\": \"a1170dba-d226-4772-89b6-9f45fb03d368\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"d3feb915-cfb4-4185-8525-9185496fce8d\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;16d19bd3-8d48-4dca-aa85-db4c9607c99f ARID=c7172e67-5da6-46b0-ac83-c1a5b5be4ded \u003e GW.Http.CallContext/c7172e67-5da6-46b0-ac83-c1a5b5be4ded \u003e GW.ExecuteQuery/d3feb915-cfb4-4185-8525-9185496fce8d \u003e DN.FE.ExecuteQuery/a1170dba-d226-4772-89b6-9f45fb03d368)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:41:46.391]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:41:43.1465223Z);\nlet _endTime = datetime(2026-08-24T21:41:43.1465227Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-sdm9vzp4vv2s'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:41:46.3064953Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;16d19bd3-8d48-4dca-aa85-db4c9607c99f\",\r\n \"activityId\": \"c7172e67-5da6-46b0-ac83-c1a5b5be4ded\",\r\n \"subActivityId\": \"a1170dba-d226-4772-89b6-9f45fb03d368\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"d3feb915-cfb4-4185-8525-9185496fce8d\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;16d19bd3-8d48-4dca-aa85-db4c9607c99f ARID=c7172e67-5da6-46b0-ac83-c1a5b5be4ded \u003e GW.Http.CallContext/c7172e67-5da6-46b0-ac83-c1a5b5be4ded \u003e GW.ExecuteQuery/d3feb915-cfb4-4185-8525-9185496fce8d \u003e DN.FE.ExecuteQuery/a1170dba-d226-4772-89b6-9f45fb03d368)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[21:41:52.877]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:41:49.6902087Z);\nlet _endTime = datetime(2026-08-24T21:41:49.6902091Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-sdm9vzp4vv2s'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:41:52.8452126Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;7ded6e3a-2d78-4f8a-bb3e-3fe5cff3f43c\",\r\n \"activityId\": \"9f24226b-f7d3-4eab-aea7-69e5e64814d8\",\r\n \"subActivityId\": \"2a442ad9-3953-4a76-a5d2-fa2895704a6b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"7111592a-ec51-4e3b-862a-f171de9c1066\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;7ded6e3a-2d78-4f8a-bb3e-3fe5cff3f43c ARID=9f24226b-f7d3-4eab-aea7-69e5e64814d8 \u003e GW.Http.CallContext/9f24226b-f7d3-4eab-aea7-69e5e64814d8 \u003e GW.ExecuteQuery/7111592a-ec51-4e3b-862a-f171de9c1066 \u003e DN.FE.ExecuteQuery/2a442ad9-3953-4a76-a5d2-fa2895704a6b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:41:53.034]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:41:49.6902087Z);\nlet _endTime = datetime(2026-08-24T21:41:49.6902091Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-sdm9vzp4vv2s'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:41:52.8452126Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;7ded6e3a-2d78-4f8a-bb3e-3fe5cff3f43c\",\r\n \"activityId\": \"9f24226b-f7d3-4eab-aea7-69e5e64814d8\",\r\n \"subActivityId\": \"2a442ad9-3953-4a76-a5d2-fa2895704a6b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"7111592a-ec51-4e3b-862a-f171de9c1066\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;7ded6e3a-2d78-4f8a-bb3e-3fe5cff3f43c ARID=9f24226b-f7d3-4eab-aea7-69e5e64814d8 \u003e GW.Http.CallContext/9f24226b-f7d3-4eab-aea7-69e5e64814d8 \u003e GW.ExecuteQuery/7111592a-ec51-4e3b-862a-f171de9c1066 \u003e DN.FE.ExecuteQuery/2a442ad9-3953-4a76-a5d2-fa2895704a6b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... cluster entered terminal Failed state after role assignment deployment 20:28:03 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.001s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.001s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ...
20:25:41	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 87c9d3c	e2e	failed	★#6662 (open)	1	e2e (1) query-basic: command failed: exit status 1 20:25:41 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 606.962s. Expected success, but got an error: <errors.errorString \| 0xc0009b8890>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[21:33:39.927]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:33:34.7399088Z);\nlet _endTime = datetime(2026-08-24T21:33:34.7399092Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8x5xzz8skd6l'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:33:39.9041454Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;3ee56433-bb81-45eb-b932-ee828be0698d\",\r\n \"activityId\": \"0b55016d-8c54-42c2-8cdc-8a328b8d6d7d\",\r\n \"subActivityId\": \"5fec2564-1b41-4e19-8b38-9f621cfecc7d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"c71f0e3b-688d-473b-837f-87b8ee173b88\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;3ee56433-bb81-45eb-b932-ee828be0698d ARID=0b55016d-8c54-42c2-8cdc-8a328b8d6d7d \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/861bdbb2-7dca-4bb1-a9cf-b7c3656731d7 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/861bdbb2-7dca-4bb1-a9cf-b7c3656731d7 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/c71f0e3b-688d-473b-837f-87b8ee173b88 \u003e DN.FE.ExecuteQuery/5fec2564-1b41-4e19-8b38-9f621cfecc7d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:33:40.031]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:33:34.7399088Z);\nlet _endTime = datetime(2026-08-24T21:33:34.7399092Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8x5xzz8skd6l'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:33:39.9041454Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;3ee56433-bb81-45eb-b932-ee828be0698d\",\r\n \"activityId\": \"0b55016d-8c54-42c2-8cdc-8a328b8d6d7d\",\r\n \"subActivityId\": \"5fec2564-1b41-4e19-8b38-9f621cfecc7d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"c71f0e3b-688d-473b-837f-87b8ee173b88\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;3ee56433-bb81-45eb-b932-ee828be0698d ARID=0b55016d-8c54-42c2-8cdc-8a328b8d6d7d \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/861bdbb2-7dca-4bb1-a9cf-b7c3656731d7 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/861bdbb2-7dca-4bb1-a9cf-b7c3656731d7 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/c71f0e3b-688d-473b-837f-87b8ee173b88 \u003e DN.FE.ExecuteQuery/5fec2564-1b41-4e19-8b38-9f621cfecc7d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[21:33:45.871]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:33:40.0938878Z);\nlet _endTime = datetime(2026-08-24T21:33:40.0938882Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8x5xzz8skd6l'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:33:45.8479456Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;96873d82-5ef0-43eb-a74d-45256ccc6d85\",\r\n \"activityId\": \"a7493440-b2f3-4e69-8824-c220555bf425\",\r\n \"subActivityId\": \"7f2f2c5e-0c26-4249-9c73-f463c8826d3d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"363a2ae3-d9d7-4e71-a139-922284c005f1\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;96873d82-5ef0-43eb-a74d-45256ccc6d85 ARID=a7493440-b2f3-4e69-8824-c220555bf425 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/cbf872ee-2e17-4232-b13d-a7e05647474a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/cbf872ee-2e17-4232-b13d-a7e05647474a \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/363a2ae3-d9d7-4e71-a139-922284c005f1 \u003e DN.FE.ExecuteQuery/7f2f2c5e-0c26-4249-9c73-f463c8826d3d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:33:45.923]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:33:40.0938878Z);\nlet _endTime = datetime(2026-08-24T21:33:40.0938882Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8x5xzz8skd6l'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:33:45.8479456Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;96873d82-5ef0-43eb-a74d-45256ccc6d85\",\r\n \"activityId\": \"a7493440-b2f3-4e69-8824-c220555bf425\",\r\n \"subActivityId\": \"7f2f2c5e-0c26-4249-9c73-f463c8826d3d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"363a2ae3-d9d7-4e71-a139-922284c005f1\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;96873d82-5ef0-43eb-a74d-45256ccc6d85 ARID=a7493440-b2f3-4e69-8824-c220555bf425 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/cbf872ee-2e17-4232-b13d-a7e05647474a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/cbf872ee-2e17-4232-b13d-a7e05647474a \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/363a2ae3-d9d7-4e71-a139-922284c005f1 \u003e DN.FE.ExecuteQuery/7f2f2c5e-0c26-4249-9c73-f463c8826d3d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[21:33:53.707]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:33:50.1150311Z);\nlet _endTime = datetime(2026-08-24T21:33:50.1150315Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8x5xzz8skd6l'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:33:53.6861063Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;2008e7e2-f66d-4bba-ab0c-fa9c9cc8792b\",\r\n \"activityId\": \"2e144aa9-1441-4208-9ef8-1602c42e9f13\",\r\n \"subActivityId\": \"8a6c32f3-9065-40d0-b597-c44fd5dbbac2\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"381df9bd-a94b-477c-879b-22493a4271de\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;2008e7e2-f66d-4bba-ab0c-fa9c9cc8792b ARID=2e144aa9-1441-4208-9ef8-1602c42e9f13 \u003e GW.Http.CallContext/2e144aa9-1441-4208-9ef8-1602c42e9f13 \u003e GW.ExecuteQuery/381df9bd-a94b-477c-879b-22493a4271de \u003e DN.FE.ExecuteQuery/8a6c32f3-9065-40d0-b597-c44fd5dbbac2)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:33:53.799]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:33:50.1150311Z);\nlet _endTime = datetime(2026-08-24T21:33:50.1150315Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8x5xzz8skd6l'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:33:53.6861063Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;2008e7e2-f66d-4bba-ab0c-fa9c9cc8792b\",\r\n \"activityId\": \"2e144aa9-1441-4208-9ef8-1602c42e9f13\",\r\n \"subActivityId\": \"8a6c32f3-9065-40d0-b597-c44fd5dbbac2\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"381df9bd-a94b-477c-879b-22493a4271de\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;2008e7e2-f66d-4bba-ab0c-fa9c9cc8792b ARID=2e144aa9-1441-4208-9ef8-1602c42e9f13 \u003e GW.Http.CallContext/2e144aa9-1441-4208-9ef8-1602c42e9f13 \u003e GW.ExecuteQuery/381df9bd-a94b-477c-879b-22493a4271de \u003e DN.FE.ExecuteQuery/8a6c32f3-9065-40d0-b597-c44fd5dbbac2)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 606.962s. Expected success, but got an error: <errors.errorString \| 0xc0009b8890>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[21:33:39.927]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:33:34.7399088Z);\nlet _endTime = datetime(2026-08-24T21:33:34.7399092Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8x5xzz8skd6l'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:33:39.9041454Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;3ee56433-bb81-45eb-b932-ee828be0698d\",\r\n \"activityId\": \"0b55016d-8c54-42c2-8cdc-8a328b8d6d7d\",\r\n \"subActivityId\": \"5fec2564-1b41-4e19-8b38-9f621cfecc7d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"c71f0e3b-688d-473b-837f-87b8ee173b88\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;3ee56433-bb81-45eb-b932-ee828be0698d ARID=0b55016d-8c54-42c2-8cdc-8a328b8d6d7d \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/861bdbb2-7dca-4bb1-a9cf-b7c3656731d7 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/861bdbb2-7dca-4bb1-a9cf-b7c3656731d7 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/c71f0e3b-688d-473b-837f-87b8ee173b88 \u003e DN.FE.ExecuteQuery/5fec2564-1b41-4e19-8b38-9f621cfecc7d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:33:40.031]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:33:34.7399088Z);\nlet _endTime = datetime(2026-08-24T21:33:34.7399092Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8x5xzz8skd6l'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:33:39.9041454Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;3ee56433-bb81-45eb-b932-ee828be0698d\",\r\n \"activityId\": \"0b55016d-8c54-42c2-8cdc-8a328b8d6d7d\",\r\n \"subActivityId\": \"5fec2564-1b41-4e19-8b38-9f621cfecc7d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"c71f0e3b-688d-473b-837f-87b8ee173b88\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;3ee56433-bb81-45eb-b932-ee828be0698d ARID=0b55016d-8c54-42c2-8cdc-8a328b8d6d7d \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/861bdbb2-7dca-4bb1-a9cf-b7c3656731d7 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/861bdbb2-7dca-4bb1-a9cf-b7c3656731d7 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/c71f0e3b-688d-473b-837f-87b8ee173b88 \u003e DN.FE.ExecuteQuery/5fec2564-1b41-4e19-8b38-9f621cfecc7d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[21:33:45.871]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:33:40.0938878Z);\nlet _endTime = datetime(2026-08-24T21:33:40.0938882Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8x5xzz8skd6l'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:33:45.8479456Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;96873d82-5ef0-43eb-a74d-45256ccc6d85\",\r\n \"activityId\": \"a7493440-b2f3-4e69-8824-c220555bf425\",\r\n \"subActivityId\": \"7f2f2c5e-0c26-4249-9c73-f463c8826d3d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"363a2ae3-d9d7-4e71-a139-922284c005f1\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;96873d82-5ef0-43eb-a74d-45256ccc6d85 ARID=a7493440-b2f3-4e69-8824-c220555bf425 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/cbf872ee-2e17-4232-b13d-a7e05647474a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/cbf872ee-2e17-4232-b13d-a7e05647474a \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/363a2ae3-d9d7-4e71-a139-922284c005f1 \u003e DN.FE.ExecuteQuery/7f2f2c5e-0c26-4249-9c73-f463c8826d3d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:33:45.923]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:33:40.0938878Z);\nlet _endTime = datetime(2026-08-24T21:33:40.0938882Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8x5xzz8skd6l'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:33:45.8479456Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;96873d82-5ef0-43eb-a74d-45256ccc6d85\",\r\n \"activityId\": \"a7493440-b2f3-4e69-8824-c220555bf425\",\r\n \"subActivityId\": \"7f2f2c5e-0c26-4249-9c73-f463c8826d3d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"363a2ae3-d9d7-4e71-a139-922284c005f1\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;96873d82-5ef0-43eb-a74d-45256ccc6d85 ARID=a7493440-b2f3-4e69-8824-c220555bf425 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/cbf872ee-2e17-4232-b13d-a7e05647474a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/cbf872ee-2e17-4232-b13d-a7e05647474a \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/363a2ae3-d9d7-4e71-a139-922284c005f1 \u003e DN.FE.ExecuteQuery/7f2f2c5e-0c26-4249-9c73-f463c8826d3d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[21:33:53.707]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:33:50.1150311Z);\nlet _endTime = datetime(2026-08-24T21:33:50.1150315Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8x5xzz8skd6l'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:33:53.6861063Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;2008e7e2-f66d-4bba-ab0c-fa9c9cc8792b\",\r\n \"activityId\": \"2e144aa9-1441-4208-9ef8-1602c42e9f13\",\r\n \"subActivityId\": \"8a6c32f3-9065-40d0-b597-c44fd5dbbac2\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"381df9bd-a94b-477c-879b-22493a4271de\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;2008e7e2-f66d-4bba-ab0c-fa9c9cc8792b ARID=2e144aa9-1441-4208-9ef8-1602c42e9f13 \u003e GW.Http.CallContext/2e144aa9-1441-4208-9ef8-1602c42e9f13 \u003e GW.ExecuteQuery/381df9bd-a94b-477c-879b-22493a4271de \u003e DN.FE.ExecuteQuery/8a6c32f3-9065-40d0-b597-c44fd5dbbac2)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:33:53.799]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:33:50.1150311Z);\nlet _endTime = datetime(2026-08-24T21:33:50.1150315Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8x5xzz8skd6l'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:33:53.6861063Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;2008e7e2-f66d-4bba-ab0c-fa9c9cc8792b\",\r\n \"activityId\": \"2e144aa9-1441-4208-9ef8-1602c42e9f13\",\r\n \"subActivityId\": \"8a6c32f3-9065-40d0-b597-c44fd5dbbac2\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"381df9bd-a94b-477c-879b-22493a4271de\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;2008e7e2-f66d-4bba-ab0c-fa9c9cc8792b ARID=2e144aa9-1441-4208-9ef8-1602c42e9f13 \u003e GW.Http.CallContext/2e144aa9-1441-4208-9ef8-1602c42e9f13 \u003e GW.ExecuteQuery/381df9bd-a94b-477c-879b-22493a4271de \u003e DN.FE.ExecuteQuery/8a6c32f3-9065-40d0-b597-c44fd5dbbac2)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ...
20:20:43	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 2a2bf6c	alert, e2e	failed	⚠#6641 (open)	5	e2e (4) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 20:20:43 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000a7afc0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-qgs92k2n68xw" to finish creating: GET https://rp.j8136960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f8d27b3b-7241-49b3-bf1d-66bcf6fbfc8a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f8d27b3b-7241-49b3-bf1d-66bcf6fbfc8a", "name": "f8d27b3b-7241-49b3-bf1d-66bcf6fbfc8a", "status": "Failed", "startTime": "2026-08-24T21:11:59.82493651Z", "endTime": "2026-08-24T21:31:07.397499058Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000a7afc0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-qgs92k2n68xw" to finish creating: GET https://rp.j8136960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f8d27b3b-7241-49b3-bf1d-66bcf6fbfc8a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f8d27b3b-7241-49b3-bf1d-66bcf6fbfc8a", "name": "f8d27b3b-7241-49b3-bf1d-66bcf6fbfc8a", "status": "Failed", "startTime": "2026-08-24T21:11:59.82493651Z", "endTime": "2026-08-24T21:31:07.397499058Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 20:20:43 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc0012a4ea0>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-26mdm7dwkvm7" to finish creating: GET https://rp.j8136960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fea58607-6b10-431a-a2a1-cc0c116e10b9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fea58607-6b10-431a-a2a1-cc0c116e10b9", "name": "fea58607-6b10-431a-a2a1-cc0c116e10b9", "status": "Failed", "startTime": "2026-08-24T21:12:09.791388432Z", "endTime": "2026-08-24T21:31:17.457093893Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc0012a4ea0>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-26mdm7dwkvm7" to finish creating: GET https://rp.j8136960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fea58607-6b10-431a-a2a1-cc0c116e10b9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fea58607-6b10-431a-a2a1-cc0c116e10b9", "name": "fea58607-6b10-431a-a2a1-cc0c116e10b9", "status": "Failed", "startTime": "2026-08-24T21:12:09.791388432Z", "endTime": "2026-08-24T21:31:17.457093893Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred query-basic: command failed: exit status 1 20:20:43 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 619.218s. Expected success, but got an error: <errors.errorString \| 0xc0010003e0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[21:37:40.862]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:37:37.2998276Z);\nlet _endTime = datetime(2026-08-24T21:37:37.2998280Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-k8dn8g4r8ltr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:37:40.8395761Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a7f2f448-ead0-4bf7-a0e6-5fcabf17d38a\",\r\n \"activityId\": \"e9aa0456-31d3-4bdf-a2f9-0cf51d841f5e\",\r\n \"subActivityId\": \"43a214bb-e28a-4d77-8a51-8eb215a63034\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"99f96206-4dd7-46c4-a9b8-10491eab64a3\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a7f2f448-ead0-4bf7-a0e6-5fcabf17d38a ARID=e9aa0456-31d3-4bdf-a2f9-0cf51d841f5e \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/07c89248-2432-450a-a970-6507b073f540 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/07c89248-2432-450a-a970-6507b073f540 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/99f96206-4dd7-46c4-a9b8-10491eab64a3 \u003e DN.FE.ExecuteQuery/43a214bb-e28a-4d77-8a51-8eb215a63034)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:37:40.916]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:37:37.2998276Z);\nlet _endTime = datetime(2026-08-24T21:37:37.2998280Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-k8dn8g4r8ltr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:37:40.8395761Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a7f2f448-ead0-4bf7-a0e6-5fcabf17d38a\",\r\n \"activityId\": \"e9aa0456-31d3-4bdf-a2f9-0cf51d841f5e\",\r\n \"subActivityId\": \"43a214bb-e28a-4d77-8a51-8eb215a63034\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"99f96206-4dd7-46c4-a9b8-10491eab64a3\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a7f2f448-ead0-4bf7-a0e6-5fcabf17d38a ARID=e9aa0456-31d3-4bdf-a2f9-0cf51d841f5e \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/07c89248-2432-450a-a970-6507b073f540 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/07c89248-2432-450a-a970-6507b073f540 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/99f96206-4dd7-46c4-a9b8-10491eab64a3 \u003e DN.FE.ExecuteQuery/43a214bb-e28a-4d77-8a51-8eb215a63034)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[21:37:44.776]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:37:40.9782699Z);\nlet _endTime = datetime(2026-08-24T21:37:40.9782702Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-k8dn8g4r8ltr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:37:44.4496384Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a6ab5682-ab15-4ffc-8f2c-903f04846256\",\r\n \"activityId\": \"41da04b9-cdd8-4f1b-aa71-799246592373\",\r\n \"subActivityId\": \"8180892d-1919-40e9-ab32-111b1441a675\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f68272df-da5c-49db-9d06-dcd9340019b6\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a6ab5682-ab15-4ffc-8f2c-903f04846256 ARID=41da04b9-cdd8-4f1b-aa71-799246592373 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/375d1995-37d2-46ea-a910-0cf7e62e840b \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/375d1995-37d2-46ea-a910-0cf7e62e840b \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/f68272df-da5c-49db-9d06-dcd9340019b6 \u003e DN.FE.ExecuteQuery/8180892d-1919-40e9-ab32-111b1441a675)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:37:45.273]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:37:40.9782699Z);\nlet _endTime = datetime(2026-08-24T21:37:40.9782702Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-k8dn8g4r8ltr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:37:44.4496384Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a6ab5682-ab15-4ffc-8f2c-903f04846256\",\r\n \"activityId\": \"41da04b9-cdd8-4f1b-aa71-799246592373\",\r\n \"subActivityId\": \"8180892d-1919-40e9-ab32-111b1441a675\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f68272df-da5c-49db-9d06-dcd9340019b6\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a6ab5682-ab15-4ffc-8f2c-903f04846256 ARID=41da04b9-cdd8-4f1b-aa71-799246592373 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/375d1995-37d2-46ea-a910-0cf7e62e840b \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/375d1995-37d2-46ea-a910-0cf7e62e840b \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/f68272df-da5c-49db-9d06-dcd9340019b6 \u003e DN.FE.ExecuteQuery/8180892d-1919-40e9-ab32-111b1441a675)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[21:37:52.048]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:37:48.7440398Z);\nlet _endTime = datetime(2026-08-24T21:37:48.7440401Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-k8dn8g4r8ltr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:37:52.0276361Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;2d45695e-bd83-4f2f-ad2d-1f2db8c593b0\",\r\n \"activityId\": \"0e05d86a-4495-4335-ba26-b5db97c01353\",\r\n \"subActivityId\": \"2afcfb00-a02e-49a7-9141-8b2e3bea0987\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"6cc0f216-19d3-43c5-b12c-5fdd754b74bf\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;2d45695e-bd83-4f2f-ad2d-1f2db8c593b0 ARID=0e05d86a-4495-4335-ba26-b5db97c01353 \u003e GW.Http.CallContext/0e05d86a-4495-4335-ba26-b5db97c01353 \u003e GW.ExecuteQuery/6cc0f216-19d3-43c5-b12c-5fdd754b74bf \u003e DN.FE.ExecuteQuery/2afcfb00-a02e-49a7-9141-8b2e3bea0987)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:37:52.603]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:37:48.7440398Z);\nlet _endTime = datetime(2026-08-24T21:37:48.7440401Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-k8dn8g4r8ltr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:37:52.0276361Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;2d45695e-bd83-4f2f-ad2d-1f2db8c593b0\",\r\n \"activityId\": \"0e05d86a-4495-4335-ba26-b5db97c01353\",\r\n \"subActivityId\": \"2afcfb00-a02e-49a7-9141-8b2e3bea0987\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"6cc0f216-19d3-43c5-b12c-5fdd754b74bf\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;2d45695e-bd83-4f2f-ad2d-1f2db8c593b0 ARID=0e05d86a-4495-4335-ba26-b5db97c01353 \u003e GW.Http.CallContext/0e05d86a-4495-4335-ba26-b5db97c01353 \u003e GW.ExecuteQuery/6cc0f216-19d3-43c5-b12c-5fdd754b74bf \u003e DN.FE.ExecuteQuery/2afcfb00-a02e-49a7-9141-8b2e3bea0987)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 619.218s. Expected success, but got an error: <errors.errorString \| 0xc0010003e0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[21:37:40.862]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:37:37.2998276Z);\nlet _endTime = datetime(2026-08-24T21:37:37.2998280Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-k8dn8g4r8ltr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:37:40.8395761Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a7f2f448-ead0-4bf7-a0e6-5fcabf17d38a\",\r\n \"activityId\": \"e9aa0456-31d3-4bdf-a2f9-0cf51d841f5e\",\r\n \"subActivityId\": \"43a214bb-e28a-4d77-8a51-8eb215a63034\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"99f96206-4dd7-46c4-a9b8-10491eab64a3\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a7f2f448-ead0-4bf7-a0e6-5fcabf17d38a ARID=e9aa0456-31d3-4bdf-a2f9-0cf51d841f5e \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/07c89248-2432-450a-a970-6507b073f540 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/07c89248-2432-450a-a970-6507b073f540 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/99f96206-4dd7-46c4-a9b8-10491eab64a3 \u003e DN.FE.ExecuteQuery/43a214bb-e28a-4d77-8a51-8eb215a63034)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:37:40.916]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:37:37.2998276Z);\nlet _endTime = datetime(2026-08-24T21:37:37.2998280Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-k8dn8g4r8ltr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:37:40.8395761Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a7f2f448-ead0-4bf7-a0e6-5fcabf17d38a\",\r\n \"activityId\": \"e9aa0456-31d3-4bdf-a2f9-0cf51d841f5e\",\r\n \"subActivityId\": \"43a214bb-e28a-4d77-8a51-8eb215a63034\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"99f96206-4dd7-46c4-a9b8-10491eab64a3\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a7f2f448-ead0-4bf7-a0e6-5fcabf17d38a ARID=e9aa0456-31d3-4bdf-a2f9-0cf51d841f5e \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/07c89248-2432-450a-a970-6507b073f540 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/07c89248-2432-450a-a970-6507b073f540 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/99f96206-4dd7-46c4-a9b8-10491eab64a3 \u003e DN.FE.ExecuteQuery/43a214bb-e28a-4d77-8a51-8eb215a63034)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[21:37:44.776]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:37:40.9782699Z);\nlet _endTime = datetime(2026-08-24T21:37:40.9782702Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-k8dn8g4r8ltr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:37:44.4496384Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a6ab5682-ab15-4ffc-8f2c-903f04846256\",\r\n \"activityId\": \"41da04b9-cdd8-4f1b-aa71-799246592373\",\r\n \"subActivityId\": \"8180892d-1919-40e9-ab32-111b1441a675\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f68272df-da5c-49db-9d06-dcd9340019b6\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a6ab5682-ab15-4ffc-8f2c-903f04846256 ARID=41da04b9-cdd8-4f1b-aa71-799246592373 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/375d1995-37d2-46ea-a910-0cf7e62e840b \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/375d1995-37d2-46ea-a910-0cf7e62e840b \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/f68272df-da5c-49db-9d06-dcd9340019b6 \u003e DN.FE.ExecuteQuery/8180892d-1919-40e9-ab32-111b1441a675)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:37:45.273]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:37:40.9782699Z);\nlet _endTime = datetime(2026-08-24T21:37:40.9782702Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-k8dn8g4r8ltr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:37:44.4496384Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a6ab5682-ab15-4ffc-8f2c-903f04846256\",\r\n \"activityId\": \"41da04b9-cdd8-4f1b-aa71-799246592373\",\r\n \"subActivityId\": \"8180892d-1919-40e9-ab32-111b1441a675\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f68272df-da5c-49db-9d06-dcd9340019b6\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a6ab5682-ab15-4ffc-8f2c-903f04846256 ARID=41da04b9-cdd8-4f1b-aa71-799246592373 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/375d1995-37d2-46ea-a910-0cf7e62e840b \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/375d1995-37d2-46ea-a910-0cf7e62e840b \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/f68272df-da5c-49db-9d06-dcd9340019b6 \u003e DN.FE.ExecuteQuery/8180892d-1919-40e9-ab32-111b1441a675)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[21:37:52.048]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:37:48.7440398Z);\nlet _endTime = datetime(2026-08-24T21:37:48.7440401Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-k8dn8g4r8ltr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:37:52.0276361Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;2d45695e-bd83-4f2f-ad2d-1f2db8c593b0\",\r\n \"activityId\": \"0e05d86a-4495-4335-ba26-b5db97c01353\",\r\n \"subActivityId\": \"2afcfb00-a02e-49a7-9141-8b2e3bea0987\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"6cc0f216-19d3-43c5-b12c-5fdd754b74bf\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;2d45695e-bd83-4f2f-ad2d-1f2db8c593b0 ARID=0e05d86a-4495-4335-ba26-b5db97c01353 \u003e GW.Http.CallContext/0e05d86a-4495-4335-ba26-b5db97c01353 \u003e GW.ExecuteQuery/6cc0f216-19d3-43c5-b12c-5fdd754b74bf \u003e DN.FE.ExecuteQuery/2afcfb00-a02e-49a7-9141-8b2e3bea0987)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:37:52.603]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:37:48.7440398Z);\nlet _endTime = datetime(2026-08-24T21:37:48.7440401Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-k8dn8g4r8ltr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:37:52.0276361Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;2d45695e-bd83-4f2f-ad2d-1f2db8c593b0\",\r\n \"activityId\": \"0e05d86a-4495-4335-ba26-b5db97c01353\",\r\n \"subActivityId\": \"2afcfb00-a02e-49a7-9141-8b2e3bea0987\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"6cc0f216-19d3-43c5-b12c-5fdd754b74bf\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;2d45695e-bd83-4f2f-ad2d-1f2db8c593b0 ARID=0e05d86a-4495-4335-ba26-b5db97c01353 \u003e GW.Http.CallContext/0e05d86a-4495-4335-ba26-b5db97c01353 \u003e GW.ExecuteQuery/6cc0f216-19d3-43c5-b12c-5fdd754b74bf \u003e DN.FE.ExecuteQuery/2afcfb00-a02e-49a7-9141-8b2e3bea0987)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 20:20:43 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc00133a200>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-qp6jkgrm7jtk" to finish creating: GET https://rp.j8136960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7d814403-8a1a-482d-afa5-63e68ef4ed09 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7d814403-8a1a-482d-afa5-63e68ef4ed09", "name": "7d814403-8a1a-482d-afa5-63e68ef4ed09", "status": "Failed", "startTime": "2026-08-24T21:13:13.654576687Z", "endTime": "2026-08-24T21:32:17.358471293Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc00133a200>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-qp6jkgrm7jtk" to finish creating: GET https://rp.j8136960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7d814403-8a1a-482d-afa5-63e68ef4ed09 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7d814403-8a1a-482d-afa5-63e68ef4ed09", "name": "7d814403-8a1a-482d-afa5-63e68ef4ed09", "status": "Failed", "startTime": "2026-08-24T21:13:13.654576687Z", "endTime": "2026-08-24T21:32:17.358471293Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 20:20:43 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-08-24T21:24:11Z Ended: 2026-08-24T21:41:12Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8136960-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd6bajvc8sdpu7eqok3bv8u764jvvj5-cilium-cluster", pod="router-67b4cfd85c-jvk8t", severity="warning" Description: Pod ocm-arohcpci01-2sd6bajvc8sdpu7eqok3bv8u764jvvj5-cilium-cluster/router-67b4cfd85c-jvk8t has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-08-24T21:24:11Z Ended: 2026-08-24T21:44:15Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8136960-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd6bqj0fesvbh7g59o57oi9i09o3vve-agg-cluster", pod="router-7cf7b968fd-ssk4r", severity="warning" Description: Pod ocm-arohcpci01-2sd6bqj0fesvbh7g59o57oi9i09o3vve-agg-cluster/router-7cf7b968fd-ssk4r has been in a non-ready state for longer than 5 minutes.
19:57:24	pull-ci-Azure-ARO-HCP-main-e2e-parallel head cbb6788	e2e	failed	★#6232 (open)	1	e2e (1) query-basic: command failed: exit status 1 19:57:24 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 615.524s. Expected success, but got an error: <errors.errorString \| 0xc0007d2820>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[20:53:53.225]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:53:49.8410583Z);\nlet _endTime = datetime(2026-08-24T20:53:49.8410588Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bzlbgl4sq6l2'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:53:53.2004335Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;0bfeb491-43ef-465f-bbbe-a0328d62f533\",\r\n \"activityId\": \"b79f0cec-39d0-4d66-a2a8-6c17ae5a7fcf\",\r\n \"subActivityId\": \"13d8a775-b3f1-4baa-9a2d-494dfc0be1b2\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"9786189c-ea06-4d5e-a5cc-f8cf3be8aa0d\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;0bfeb491-43ef-465f-bbbe-a0328d62f533 ARID=b79f0cec-39d0-4d66-a2a8-6c17ae5a7fcf \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/2e9ea8ca-6016-4ebf-b108-9558f32002d5 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/2e9ea8ca-6016-4ebf-b108-9558f32002d5 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/9786189c-ea06-4d5e-a5cc-f8cf3be8aa0d \u003e DN.FE.ExecuteQuery/13d8a775-b3f1-4baa-9a2d-494dfc0be1b2)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:53:53.288]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:53:49.8410583Z);\nlet _endTime = datetime(2026-08-24T20:53:49.8410588Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bzlbgl4sq6l2'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:53:53.2004335Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;0bfeb491-43ef-465f-bbbe-a0328d62f533\",\r\n \"activityId\": \"b79f0cec-39d0-4d66-a2a8-6c17ae5a7fcf\",\r\n \"subActivityId\": \"13d8a775-b3f1-4baa-9a2d-494dfc0be1b2\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"9786189c-ea06-4d5e-a5cc-f8cf3be8aa0d\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;0bfeb491-43ef-465f-bbbe-a0328d62f533 ARID=b79f0cec-39d0-4d66-a2a8-6c17ae5a7fcf \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/2e9ea8ca-6016-4ebf-b108-9558f32002d5 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/2e9ea8ca-6016-4ebf-b108-9558f32002d5 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/9786189c-ea06-4d5e-a5cc-f8cf3be8aa0d \u003e DN.FE.ExecuteQuery/13d8a775-b3f1-4baa-9a2d-494dfc0be1b2)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[20:53:56.484]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:53:53.3762355Z);\nlet _endTime = datetime(2026-08-24T20:53:53.3762363Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bzlbgl4sq6l2'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:53:56.4596189Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;50c48c80-ca0d-4a4d-870a-71c254e75198\",\r\n \"activityId\": \"95683e67-f053-4393-99b1-71151c466434\",\r\n \"subActivityId\": \"661413b7-6319-4d4e-84ca-2ab054915379\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"6ac9b874-fb05-4d4b-806e-e7d9f8496026\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;50c48c80-ca0d-4a4d-870a-71c254e75198 ARID=95683e67-f053-4393-99b1-71151c466434 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/5f3695e3-ec59-4fef-9596-926ec623c669 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/5f3695e3-ec59-4fef-9596-926ec623c669 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/6ac9b874-fb05-4d4b-806e-e7d9f8496026 \u003e DN.FE.ExecuteQuery/661413b7-6319-4d4e-84ca-2ab054915379)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:53:56.541]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:53:53.3762355Z);\nlet _endTime = datetime(2026-08-24T20:53:53.3762363Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bzlbgl4sq6l2'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:53:56.4596189Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;50c48c80-ca0d-4a4d-870a-71c254e75198\",\r\n \"activityId\": \"95683e67-f053-4393-99b1-71151c466434\",\r\n \"subActivityId\": \"661413b7-6319-4d4e-84ca-2ab054915379\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"6ac9b874-fb05-4d4b-806e-e7d9f8496026\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;50c48c80-ca0d-4a4d-870a-71c254e75198 ARID=95683e67-f053-4393-99b1-71151c466434 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/5f3695e3-ec59-4fef-9596-926ec623c669 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/5f3695e3-ec59-4fef-9596-926ec623c669 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/6ac9b874-fb05-4d4b-806e-e7d9f8496026 \u003e DN.FE.ExecuteQuery/661413b7-6319-4d4e-84ca-2ab054915379)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[20:54:03.095]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:53:59.8371044Z);\nlet _endTime = datetime(2026-08-24T20:53:59.8371051Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bzlbgl4sq6l2'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:54:03.0699661Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;535f349f-0f02-4848-ad54-a4b9b368c05a\",\r\n \"activityId\": \"b8e19f30-0aeb-4bae-aa9a-2994c97ca8e2\",\r\n \"subActivityId\": \"c7079a92-9d3f-43af-ae10-33c5d509b1db\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"fe309183-9dd8-40b4-8fca-7e5ee982bcdd\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;535f349f-0f02-4848-ad54-a4b9b368c05a ARID=b8e19f30-0aeb-4bae-aa9a-2994c97ca8e2 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/30e8047d-b899-4df8-96bf-7f67a7ea3a80 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/30e8047d-b899-4df8-96bf-7f67a7ea3a80 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/fe309183-9dd8-40b4-8fca-7e5ee982bcdd \u003e DN.FE.ExecuteQuery/c7079a92-9d3f-43af-ae10-33c5d509b1db)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:54:03.206]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:53:59.8371044Z);\nlet _endTime = datetime(2026-08-24T20:53:59.8371051Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bzlbgl4sq6l2'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:54:03.0699661Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;535f349f-0f02-4848-ad54-a4b9b368c05a\",\r\n \"activityId\": \"b8e19f30-0aeb-4bae-aa9a-2994c97ca8e2\",\r\n \"subActivityId\": \"c7079a92-9d3f-43af-ae10-33c5d509b1db\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"fe309183-9dd8-40b4-8fca-7e5ee982bcdd\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;535f349f-0f02-4848-ad54-a4b9b368c05a ARID=b8e19f30-0aeb-4bae-aa9a-2994c97ca8e2 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/30e8047d-b899-4df8-96bf-7f67a7ea3a80 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/30e8047d-b899-4df8-96bf-7f67a7ea3a80 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/fe309183-9dd8-40b4-8fca-7e5ee982bcdd \u003e DN.FE.ExecuteQuery/c7079a92-9d3f-43af-ae10-33c5d509b1db)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 615.524s. Expected success, but got an error: <errors.errorString \| 0xc0007d2820>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[20:53:53.225]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:53:49.8410583Z);\nlet _endTime = datetime(2026-08-24T20:53:49.8410588Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bzlbgl4sq6l2'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:53:53.2004335Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;0bfeb491-43ef-465f-bbbe-a0328d62f533\",\r\n \"activityId\": \"b79f0cec-39d0-4d66-a2a8-6c17ae5a7fcf\",\r\n \"subActivityId\": \"13d8a775-b3f1-4baa-9a2d-494dfc0be1b2\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"9786189c-ea06-4d5e-a5cc-f8cf3be8aa0d\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;0bfeb491-43ef-465f-bbbe-a0328d62f533 ARID=b79f0cec-39d0-4d66-a2a8-6c17ae5a7fcf \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/2e9ea8ca-6016-4ebf-b108-9558f32002d5 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/2e9ea8ca-6016-4ebf-b108-9558f32002d5 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/9786189c-ea06-4d5e-a5cc-f8cf3be8aa0d \u003e DN.FE.ExecuteQuery/13d8a775-b3f1-4baa-9a2d-494dfc0be1b2)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:53:53.288]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:53:49.8410583Z);\nlet _endTime = datetime(2026-08-24T20:53:49.8410588Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bzlbgl4sq6l2'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:53:53.2004335Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;0bfeb491-43ef-465f-bbbe-a0328d62f533\",\r\n \"activityId\": \"b79f0cec-39d0-4d66-a2a8-6c17ae5a7fcf\",\r\n \"subActivityId\": \"13d8a775-b3f1-4baa-9a2d-494dfc0be1b2\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"9786189c-ea06-4d5e-a5cc-f8cf3be8aa0d\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;0bfeb491-43ef-465f-bbbe-a0328d62f533 ARID=b79f0cec-39d0-4d66-a2a8-6c17ae5a7fcf \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/2e9ea8ca-6016-4ebf-b108-9558f32002d5 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/2e9ea8ca-6016-4ebf-b108-9558f32002d5 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/9786189c-ea06-4d5e-a5cc-f8cf3be8aa0d \u003e DN.FE.ExecuteQuery/13d8a775-b3f1-4baa-9a2d-494dfc0be1b2)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[20:53:56.484]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:53:53.3762355Z);\nlet _endTime = datetime(2026-08-24T20:53:53.3762363Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bzlbgl4sq6l2'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:53:56.4596189Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;50c48c80-ca0d-4a4d-870a-71c254e75198\",\r\n \"activityId\": \"95683e67-f053-4393-99b1-71151c466434\",\r\n \"subActivityId\": \"661413b7-6319-4d4e-84ca-2ab054915379\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"6ac9b874-fb05-4d4b-806e-e7d9f8496026\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;50c48c80-ca0d-4a4d-870a-71c254e75198 ARID=95683e67-f053-4393-99b1-71151c466434 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/5f3695e3-ec59-4fef-9596-926ec623c669 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/5f3695e3-ec59-4fef-9596-926ec623c669 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/6ac9b874-fb05-4d4b-806e-e7d9f8496026 \u003e DN.FE.ExecuteQuery/661413b7-6319-4d4e-84ca-2ab054915379)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:53:56.541]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:53:53.3762355Z);\nlet _endTime = datetime(2026-08-24T20:53:53.3762363Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bzlbgl4sq6l2'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:53:56.4596189Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;50c48c80-ca0d-4a4d-870a-71c254e75198\",\r\n \"activityId\": \"95683e67-f053-4393-99b1-71151c466434\",\r\n \"subActivityId\": \"661413b7-6319-4d4e-84ca-2ab054915379\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"6ac9b874-fb05-4d4b-806e-e7d9f8496026\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;50c48c80-ca0d-4a4d-870a-71c254e75198 ARID=95683e67-f053-4393-99b1-71151c466434 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/5f3695e3-ec59-4fef-9596-926ec623c669 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/5f3695e3-ec59-4fef-9596-926ec623c669 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/6ac9b874-fb05-4d4b-806e-e7d9f8496026 \u003e DN.FE.ExecuteQuery/661413b7-6319-4d4e-84ca-2ab054915379)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[20:54:03.095]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:53:59.8371044Z);\nlet _endTime = datetime(2026-08-24T20:53:59.8371051Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bzlbgl4sq6l2'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:54:03.0699661Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;535f349f-0f02-4848-ad54-a4b9b368c05a\",\r\n \"activityId\": \"b8e19f30-0aeb-4bae-aa9a-2994c97ca8e2\",\r\n \"subActivityId\": \"c7079a92-9d3f-43af-ae10-33c5d509b1db\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"fe309183-9dd8-40b4-8fca-7e5ee982bcdd\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;535f349f-0f02-4848-ad54-a4b9b368c05a ARID=b8e19f30-0aeb-4bae-aa9a-2994c97ca8e2 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/30e8047d-b899-4df8-96bf-7f67a7ea3a80 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/30e8047d-b899-4df8-96bf-7f67a7ea3a80 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/fe309183-9dd8-40b4-8fca-7e5ee982bcdd \u003e DN.FE.ExecuteQuery/c7079a92-9d3f-43af-ae10-33c5d509b1db)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:54:03.206]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:53:59.8371044Z);\nlet _endTime = datetime(2026-08-24T20:53:59.8371051Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bzlbgl4sq6l2'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:54:03.0699661Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;535f349f-0f02-4848-ad54-a4b9b368c05a\",\r\n \"activityId\": \"b8e19f30-0aeb-4bae-aa9a-2994c97ca8e2\",\r\n \"subActivityId\": \"c7079a92-9d3f-43af-ae10-33c5d509b1db\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"fe309183-9dd8-40b4-8fca-7e5ee982bcdd\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;535f349f-0f02-4848-ad54-a4b9b368c05a ARID=b8e19f30-0aeb-4bae-aa9a-2994c97ca8e2 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/30e8047d-b899-4df8-96bf-7f67a7ea3a80 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/30e8047d-b899-4df8-96bf-7f67a7ea3a80 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/fe309183-9dd8-40b4-8fca-7e5ee982bcdd \u003e DN.FE.ExecuteQuery/c7079a92-9d3f-43af-ae10-33c5d509b1db)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ...
19:54:38	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 9906d38	e2e	failed	★#6658 (open)	2	e2e (2) ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 19:54:38 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:166]: failed to create node pool npupgrade-4-20 with version 4.20.34 Unexpected error: <fmt.wrapError \| 0xc00031f900>: failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-tczrqg" in resourcegroup="rg-np-version-upgrade-tczrqg-fhlnbg67qjlz" to finish creating: GET https://rp.j7254272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c2d7b313-a7f0-40de-93b0-918f61dd2377 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c2d7b313-a7f0-40de-93b0-918f61dd2377", "name": "c2d7b313-a7f0-40de-93b0-918f61dd2377", "status": "Failed", "startTime": "2026-08-24T20:58:35.24134299Z", "endTime": "2026-08-24T21:17:39.53487523Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:166]: failed to create node pool npupgrade-4-20 with version 4.20.34 Unexpected error: <fmt.wrapError \| 0xc00031f900>: failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-tczrqg" in resourcegroup="rg-np-version-upgrade-tczrqg-fhlnbg67qjlz" to finish creating: GET https://rp.j7254272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c2d7b313-a7f0-40de-93b0-918f61dd2377 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c2d7b313-a7f0-40de-93b0-918f61dd2377", "name": "c2d7b313-a7f0-40de-93b0-918f61dd2377", "status": "Failed", "startTime": "2026-08-24T20:58:35.24134299Z", "endTime": "2026-08-24T21:17:39.53487523Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred query-basic: command failed: exit status 1 19:54:38 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 603.830s. Expected success, but got an error: <errors.errorString \| 0xc0004931d0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[21:15:16.025]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:15:11.3931787Z);\nlet _endTime = datetime(2026-08-24T21:15:11.3931797Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8xqfrdjbw4lb'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:15:16.0014048Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;f906d12a-14d5-4726-b29d-abd04f327f3a\",\r\n \"activityId\": \"2e49c6f6-575c-42f6-b8f3-dad7f49c8a1e\",\r\n \"subActivityId\": \"aa8f470b-0b53-475d-a4fa-cf95cf2df525\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"c446787f-4208-43bb-af92-7f3c5bab6df5\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;f906d12a-14d5-4726-b29d-abd04f327f3a ARID=2e49c6f6-575c-42f6-b8f3-dad7f49c8a1e \u003e GW.Http.CallContext/2e49c6f6-575c-42f6-b8f3-dad7f49c8a1e \u003e GW.ExecuteQuery/c446787f-4208-43bb-af92-7f3c5bab6df5 \u003e DN.FE.ExecuteQuery/aa8f470b-0b53-475d-a4fa-cf95cf2df525)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:15:16.083]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:15:11.3931787Z);\nlet _endTime = datetime(2026-08-24T21:15:11.3931797Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8xqfrdjbw4lb'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:15:16.0014048Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;f906d12a-14d5-4726-b29d-abd04f327f3a\",\r\n \"activityId\": \"2e49c6f6-575c-42f6-b8f3-dad7f49c8a1e\",\r\n \"subActivityId\": \"aa8f470b-0b53-475d-a4fa-cf95cf2df525\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"c446787f-4208-43bb-af92-7f3c5bab6df5\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;f906d12a-14d5-4726-b29d-abd04f327f3a ARID=2e49c6f6-575c-42f6-b8f3-dad7f49c8a1e \u003e GW.Http.CallContext/2e49c6f6-575c-42f6-b8f3-dad7f49c8a1e \u003e GW.ExecuteQuery/c446787f-4208-43bb-af92-7f3c5bab6df5 \u003e DN.FE.ExecuteQuery/aa8f470b-0b53-475d-a4fa-cf95cf2df525)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[21:15:19.746]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:15:16.2075639Z);\nlet _endTime = datetime(2026-08-24T21:15:16.2075645Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8xqfrdjbw4lb'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:15:19.7247115Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;52787429-40fb-41f6-897a-171e6f45b558\",\r\n \"activityId\": \"451ccf71-58ba-4e87-941d-9a32337dc0a7\",\r\n \"subActivityId\": \"fe044149-3b5c-44f8-a307-080742dbb5ff\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f8c25e56-5afe-4b10-bd5d-0dd97d7b4890\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;52787429-40fb-41f6-897a-171e6f45b558 ARID=451ccf71-58ba-4e87-941d-9a32337dc0a7 \u003e GW.Http.CallContext/451ccf71-58ba-4e87-941d-9a32337dc0a7 \u003e GW.ExecuteQuery/f8c25e56-5afe-4b10-bd5d-0dd97d7b4890 \u003e DN.FE.ExecuteQuery/fe044149-3b5c-44f8-a307-080742dbb5ff)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:15:19.746]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "context canceled", "name": "cosmosResourceSnapshots" }�[0m �[37m[21:15:19.814]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:15:16.2075639Z);\nlet _endTime = datetime(2026-08-24T21:15:16.2075645Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8xqfrdjbw4lb'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:15:19.7247115Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;52787429-40fb-41f6-897a-171e6f45b558\",\r\n \"activityId\": \"451ccf71-58ba-4e87-941d-9a32337dc0a7\",\r\n \"subActivityId\": \"fe044149-3b5c-44f8-a307-080742dbb5ff\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f8c25e56-5afe-4b10-bd5d-0dd97d7b4890\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;52787429-40fb-41f6-897a-171e6f45b558 ARID=451ccf71-58ba-4e87-941d-9a32337dc0a7 \u003e GW.Http.CallContext/451ccf71-58ba-4e87-941d-9a32337dc0a7 \u003e GW.ExecuteQuery/f8c25e56-5afe-4b10-bd5d-0dd97d7b4890 \u003e DN.FE.ExecuteQuery/fe044149-3b5c-44f8-a307-080742dbb5ff)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[21:15:27.083]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:15:23.4959268Z);\nlet _endTime = datetime(2026-08-24T21:15:23.4959274Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8xqfrdjbw4lb'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:15:27.0563191Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a50f7a18-f495-4b29-82b8-dc128132d94e\",\r\n \"activityId\": \"c2e33394-53e9-4db1-b207-3f75282af04f\",\r\n \"subActivityId\": \"869609d0-5465-4389-80a4-03eb382e7d30\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"b0dc1001-d29d-4093-9f9a-e8d6af8ae3e9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a50f7a18-f495-4b29-82b8-dc128132d94e ARID=c2e33394-53e9-4db1-b207-3f75282af04f \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/408a6271-9bb4-47a9-92f1-c223c7195aca \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/408a6271-9bb4-47a9-92f1-c223c7195aca \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/b0dc1001-d29d-4093-9f9a-e8d6af8ae3e9 \u003e DN.FE.ExecuteQuery/869609d0-5465-4389-80a4-03eb382e7d30)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:15:27.198]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:15:23.4959268Z);\nlet _endTime = datetime(2026-08-24T21:15:23.4959274Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8xqfrdjbw4lb'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:15:27.0563191Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a50f7a18-f495-4b29-82b8-dc128132d94e\",\r\n \"activityId\": \"c2e33394-53e9-4db1-b207-3f75282af04f\",\r\n \"subActivityId\": \"869609d0-5465-4389-80a4-03eb382e7d30\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"b0dc1001-d29d-4093-9f9a-e8d6af8ae3e9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a50f7a18-f495-4b29-82b8-dc128132d94e ARID=c2e33394-53e9-4db1-b207-3f75282af04f \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/408a6271-9bb4-47a9-92f1-c223c7195aca \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/408a6271-9bb4-47a9-92f1-c223c7195aca \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/b0dc1001-d29d-4093-9f9a-e8d6af8ae3e9 \u003e DN.FE.ExecuteQuery/869609d0-5465-4389-80a4-03eb382e7d30)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 603.830s. Expected success, but got an error: <errors.errorString \| 0xc0004931d0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[21:15:16.025]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:15:11.3931787Z);\nlet _endTime = datetime(2026-08-24T21:15:11.3931797Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8xqfrdjbw4lb'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:15:16.0014048Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;f906d12a-14d5-4726-b29d-abd04f327f3a\",\r\n \"activityId\": \"2e49c6f6-575c-42f6-b8f3-dad7f49c8a1e\",\r\n \"subActivityId\": \"aa8f470b-0b53-475d-a4fa-cf95cf2df525\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"c446787f-4208-43bb-af92-7f3c5bab6df5\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;f906d12a-14d5-4726-b29d-abd04f327f3a ARID=2e49c6f6-575c-42f6-b8f3-dad7f49c8a1e \u003e GW.Http.CallContext/2e49c6f6-575c-42f6-b8f3-dad7f49c8a1e \u003e GW.ExecuteQuery/c446787f-4208-43bb-af92-7f3c5bab6df5 \u003e DN.FE.ExecuteQuery/aa8f470b-0b53-475d-a4fa-cf95cf2df525)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:15:16.083]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:15:11.3931787Z);\nlet _endTime = datetime(2026-08-24T21:15:11.3931797Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8xqfrdjbw4lb'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:15:16.0014048Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;f906d12a-14d5-4726-b29d-abd04f327f3a\",\r\n \"activityId\": \"2e49c6f6-575c-42f6-b8f3-dad7f49c8a1e\",\r\n \"subActivityId\": \"aa8f470b-0b53-475d-a4fa-cf95cf2df525\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"c446787f-4208-43bb-af92-7f3c5bab6df5\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;f906d12a-14d5-4726-b29d-abd04f327f3a ARID=2e49c6f6-575c-42f6-b8f3-dad7f49c8a1e \u003e GW.Http.CallContext/2e49c6f6-575c-42f6-b8f3-dad7f49c8a1e \u003e GW.ExecuteQuery/c446787f-4208-43bb-af92-7f3c5bab6df5 \u003e DN.FE.ExecuteQuery/aa8f470b-0b53-475d-a4fa-cf95cf2df525)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[21:15:19.746]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:15:16.2075639Z);\nlet _endTime = datetime(2026-08-24T21:15:16.2075645Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8xqfrdjbw4lb'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:15:19.7247115Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;52787429-40fb-41f6-897a-171e6f45b558\",\r\n \"activityId\": \"451ccf71-58ba-4e87-941d-9a32337dc0a7\",\r\n \"subActivityId\": \"fe044149-3b5c-44f8-a307-080742dbb5ff\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f8c25e56-5afe-4b10-bd5d-0dd97d7b4890\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;52787429-40fb-41f6-897a-171e6f45b558 ARID=451ccf71-58ba-4e87-941d-9a32337dc0a7 \u003e GW.Http.CallContext/451ccf71-58ba-4e87-941d-9a32337dc0a7 \u003e GW.ExecuteQuery/f8c25e56-5afe-4b10-bd5d-0dd97d7b4890 \u003e DN.FE.ExecuteQuery/fe044149-3b5c-44f8-a307-080742dbb5ff)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:15:19.746]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "context canceled", "name": "cosmosResourceSnapshots" }�[0m �[37m[21:15:19.814]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:15:16.2075639Z);\nlet _endTime = datetime(2026-08-24T21:15:16.2075645Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8xqfrdjbw4lb'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:15:19.7247115Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;52787429-40fb-41f6-897a-171e6f45b558\",\r\n \"activityId\": \"451ccf71-58ba-4e87-941d-9a32337dc0a7\",\r\n \"subActivityId\": \"fe044149-3b5c-44f8-a307-080742dbb5ff\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f8c25e56-5afe-4b10-bd5d-0dd97d7b4890\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;52787429-40fb-41f6-897a-171e6f45b558 ARID=451ccf71-58ba-4e87-941d-9a32337dc0a7 \u003e GW.Http.CallContext/451ccf71-58ba-4e87-941d-9a32337dc0a7 \u003e GW.ExecuteQuery/f8c25e56-5afe-4b10-bd5d-0dd97d7b4890 \u003e DN.FE.ExecuteQuery/fe044149-3b5c-44f8-a307-080742dbb5ff)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[21:15:27.083]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:15:23.4959268Z);\nlet _endTime = datetime(2026-08-24T21:15:23.4959274Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8xqfrdjbw4lb'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:15:27.0563191Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a50f7a18-f495-4b29-82b8-dc128132d94e\",\r\n \"activityId\": \"c2e33394-53e9-4db1-b207-3f75282af04f\",\r\n \"subActivityId\": \"869609d0-5465-4389-80a4-03eb382e7d30\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"b0dc1001-d29d-4093-9f9a-e8d6af8ae3e9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a50f7a18-f495-4b29-82b8-dc128132d94e ARID=c2e33394-53e9-4db1-b207-3f75282af04f \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/408a6271-9bb4-47a9-92f1-c223c7195aca \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/408a6271-9bb4-47a9-92f1-c223c7195aca \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/b0dc1001-d29d-4093-9f9a-e8d6af8ae3e9 \u003e DN.FE.ExecuteQuery/869609d0-5465-4389-80a4-03eb382e7d30)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:15:27.198]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:15:23.4959268Z);\nlet _endTime = datetime(2026-08-24T21:15:23.4959274Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8xqfrdjbw4lb'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:15:27.0563191Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a50f7a18-f495-4b29-82b8-dc128132d94e\",\r\n \"activityId\": \"c2e33394-53e9-4db1-b207-3f75282af04f\",\r\n \"subActivityId\": \"869609d0-5465-4389-80a4-03eb382e7d30\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"b0dc1001-d29d-4093-9f9a-e8d6af8ae3e9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a50f7a18-f495-4b29-82b8-dc128132d94e ARID=c2e33394-53e9-4db1-b207-3f75282af04f \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/408a6271-9bb4-47a9-92f1-c223c7195aca \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/408a6271-9bb4-47a9-92f1-c223c7195aca \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/b0dc1001-d29d-4093-9f9a-e8d6af8ae3e9 \u003e DN.FE.ExecuteQuery/869609d0-5465-4389-80a4-03eb382e7d30)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ...
19:54:36	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c6d4678	e2e	failed	★#6629 (open)	2	e2e (2) query-basic: command failed: exit status 1 19:54:36 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.008s. Expected success, but got an error: <errors.errorString \| 0xc0018343f0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[21:11:48.168]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:11:45.1785302Z);\nlet _endTime = datetime(2026-08-24T21:11:45.1785306Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-nhrxvm5llfcn'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:11:48.1484338Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;ae0286df-584b-4a34-9f74-dcee83455e56\",\r\n \"activityId\": \"5bf3a5e5-fd68-4ea3-9729-39d43d93c8ec\",\r\n \"subActivityId\": \"dfac13ec-813c-4196-8cc2-2e9a57e63812\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"101755fc-9d7c-424e-906e-dab6c556873c\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;ae0286df-584b-4a34-9f74-dcee83455e56 ARID=5bf3a5e5-fd68-4ea3-9729-39d43d93c8ec \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b44f371f-d679-44e7-8b17-97d091eb055d \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b44f371f-d679-44e7-8b17-97d091eb055d \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/101755fc-9d7c-424e-906e-dab6c556873c \u003e DN.FE.ExecuteQuery/dfac13ec-813c-4196-8cc2-2e9a57e63812)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:11:48.227]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:11:45.1785302Z);\nlet _endTime = datetime(2026-08-24T21:11:45.1785306Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-nhrxvm5llfcn'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:11:48.1484338Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;ae0286df-584b-4a34-9f74-dcee83455e56\",\r\n \"activityId\": \"5bf3a5e5-fd68-4ea3-9729-39d43d93c8ec\",\r\n \"subActivityId\": \"dfac13ec-813c-4196-8cc2-2e9a57e63812\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"101755fc-9d7c-424e-906e-dab6c556873c\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;ae0286df-584b-4a34-9f74-dcee83455e56 ARID=5bf3a5e5-fd68-4ea3-9729-39d43d93c8ec \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b44f371f-d679-44e7-8b17-97d091eb055d \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b44f371f-d679-44e7-8b17-97d091eb055d \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/101755fc-9d7c-424e-906e-dab6c556873c \u003e DN.FE.ExecuteQuery/dfac13ec-813c-4196-8cc2-2e9a57e63812)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[21:11:51.205]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:11:48.3065804Z);\nlet _endTime = datetime(2026-08-24T21:11:48.3065811Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-nhrxvm5llfcn'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:11:51.1836113Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;ee12add3-3beb-40dc-9f72-af09feb40f1e\",\r\n \"activityId\": \"aabba74b-59ad-44c8-9241-3e6c721648d8\",\r\n \"subActivityId\": \"4b05f656-fa05-436e-8ee6-fa3cad2e5460\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"2c8d9f77-bbab-4efe-829b-2617c0d7469a\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;ee12add3-3beb-40dc-9f72-af09feb40f1e ARID=aabba74b-59ad-44c8-9241-3e6c721648d8 \u003e GW.Http.CallContext/aabba74b-59ad-44c8-9241-3e6c721648d8 \u003e GW.ExecuteQuery/2c8d9f77-bbab-4efe-829b-2617c0d7469a \u003e DN.FE.ExecuteQuery/4b05f656-fa05-436e-8ee6-fa3cad2e5460)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:11:51.256]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:11:48.3065804Z);\nlet _endTime = datetime(2026-08-24T21:11:48.3065811Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-nhrxvm5llfcn'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:11:51.1836113Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;ee12add3-3beb-40dc-9f72-af09feb40f1e\",\r\n \"activityId\": \"aabba74b-59ad-44c8-9241-3e6c721648d8\",\r\n \"subActivityId\": \"4b05f656-fa05-436e-8ee6-fa3cad2e5460\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"2c8d9f77-bbab-4efe-829b-2617c0d7469a\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;ee12add3-3beb-40dc-9f72-af09feb40f1e ARID=aabba74b-59ad-44c8-9241-3e6c721648d8 \u003e GW.Http.CallContext/aabba74b-59ad-44c8-9241-3e6c721648d8 \u003e GW.ExecuteQuery/2c8d9f77-bbab-4efe-829b-2617c0d7469a \u003e DN.FE.ExecuteQuery/4b05f656-fa05-436e-8ee6-fa3cad2e5460)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[21:11:57.885]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:11:54.6424073Z);\nlet _endTime = datetime(2026-08-24T21:11:54.6424077Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-nhrxvm5llfcn'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:11:57.8650613Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;c2c9a2ce-5b50-4831-9308-2de17897e7e8\",\r\n \"activityId\": \"61421d75-0512-490c-ae8e-ed449f7b4ef1\",\r\n \"subActivityId\": \"cddb715f-ae58-4ad3-8057-d1e4ab4131aa\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"af3adfb7-44fa-4141-826e-04cfbf96ab82\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;c2c9a2ce-5b50-4831-9308-2de17897e7e8 ARID=61421d75-0512-490c-ae8e-ed449f7b4ef1 \u003e GW.Http.CallContext/61421d75-0512-490c-ae8e-ed449f7b4ef1 \u003e GW.ExecuteQuery/af3adfb7-44fa-4141-826e-04cfbf96ab82 \u003e DN.FE.ExecuteQuery/cddb715f-ae58-4ad3-8057-d1e4ab4131aa)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:11:57.986]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:11:54.6424073Z);\nlet _endTime = datetime(2026-08-24T21:11:54.6424077Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-nhrxvm5llfcn'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:11:57.8650613Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;c2c9a2ce-5b50-4831-9308-2de17897e7e8\",\r\n \"activityId\": \"61421d75-0512-490c-ae8e-ed449f7b4ef1\",\r\n \"subActivityId\": \"cddb715f-ae58-4ad3-8057-d1e4ab4131aa\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"af3adfb7-44fa-4141-826e-04cfbf96ab82\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;c2c9a2ce-5b50-4831-9308-2de17897e7e8 ARID=61421d75-0512-490c-ae8e-ed449f7b4ef1 \u003e GW.Http.CallContext/61421d75-0512-490c-ae8e-ed449f7b4ef1 \u003e GW.ExecuteQuery/af3adfb7-44fa-4141-826e-04cfbf96ab82 \u003e DN.FE.ExecuteQuery/cddb715f-ae58-4ad3-8057-d1e4ab4131aa)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.008s. Expected success, but got an error: <errors.errorString \| 0xc0018343f0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[21:11:48.168]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:11:45.1785302Z);\nlet _endTime = datetime(2026-08-24T21:11:45.1785306Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-nhrxvm5llfcn'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:11:48.1484338Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;ae0286df-584b-4a34-9f74-dcee83455e56\",\r\n \"activityId\": \"5bf3a5e5-fd68-4ea3-9729-39d43d93c8ec\",\r\n \"subActivityId\": \"dfac13ec-813c-4196-8cc2-2e9a57e63812\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"101755fc-9d7c-424e-906e-dab6c556873c\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;ae0286df-584b-4a34-9f74-dcee83455e56 ARID=5bf3a5e5-fd68-4ea3-9729-39d43d93c8ec \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b44f371f-d679-44e7-8b17-97d091eb055d \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b44f371f-d679-44e7-8b17-97d091eb055d \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/101755fc-9d7c-424e-906e-dab6c556873c \u003e DN.FE.ExecuteQuery/dfac13ec-813c-4196-8cc2-2e9a57e63812)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:11:48.227]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:11:45.1785302Z);\nlet _endTime = datetime(2026-08-24T21:11:45.1785306Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-nhrxvm5llfcn'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:11:48.1484338Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;ae0286df-584b-4a34-9f74-dcee83455e56\",\r\n \"activityId\": \"5bf3a5e5-fd68-4ea3-9729-39d43d93c8ec\",\r\n \"subActivityId\": \"dfac13ec-813c-4196-8cc2-2e9a57e63812\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"101755fc-9d7c-424e-906e-dab6c556873c\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;ae0286df-584b-4a34-9f74-dcee83455e56 ARID=5bf3a5e5-fd68-4ea3-9729-39d43d93c8ec \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b44f371f-d679-44e7-8b17-97d091eb055d \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b44f371f-d679-44e7-8b17-97d091eb055d \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/101755fc-9d7c-424e-906e-dab6c556873c \u003e DN.FE.ExecuteQuery/dfac13ec-813c-4196-8cc2-2e9a57e63812)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[21:11:51.205]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:11:48.3065804Z);\nlet _endTime = datetime(2026-08-24T21:11:48.3065811Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-nhrxvm5llfcn'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:11:51.1836113Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;ee12add3-3beb-40dc-9f72-af09feb40f1e\",\r\n \"activityId\": \"aabba74b-59ad-44c8-9241-3e6c721648d8\",\r\n \"subActivityId\": \"4b05f656-fa05-436e-8ee6-fa3cad2e5460\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"2c8d9f77-bbab-4efe-829b-2617c0d7469a\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;ee12add3-3beb-40dc-9f72-af09feb40f1e ARID=aabba74b-59ad-44c8-9241-3e6c721648d8 \u003e GW.Http.CallContext/aabba74b-59ad-44c8-9241-3e6c721648d8 \u003e GW.ExecuteQuery/2c8d9f77-bbab-4efe-829b-2617c0d7469a \u003e DN.FE.ExecuteQuery/4b05f656-fa05-436e-8ee6-fa3cad2e5460)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:11:51.256]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:11:48.3065804Z);\nlet _endTime = datetime(2026-08-24T21:11:48.3065811Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-nhrxvm5llfcn'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:11:51.1836113Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;ee12add3-3beb-40dc-9f72-af09feb40f1e\",\r\n \"activityId\": \"aabba74b-59ad-44c8-9241-3e6c721648d8\",\r\n \"subActivityId\": \"4b05f656-fa05-436e-8ee6-fa3cad2e5460\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"2c8d9f77-bbab-4efe-829b-2617c0d7469a\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;ee12add3-3beb-40dc-9f72-af09feb40f1e ARID=aabba74b-59ad-44c8-9241-3e6c721648d8 \u003e GW.Http.CallContext/aabba74b-59ad-44c8-9241-3e6c721648d8 \u003e GW.ExecuteQuery/2c8d9f77-bbab-4efe-829b-2617c0d7469a \u003e DN.FE.ExecuteQuery/4b05f656-fa05-436e-8ee6-fa3cad2e5460)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[21:11:57.885]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:11:54.6424073Z);\nlet _endTime = datetime(2026-08-24T21:11:54.6424077Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-nhrxvm5llfcn'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:11:57.8650613Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;c2c9a2ce-5b50-4831-9308-2de17897e7e8\",\r\n \"activityId\": \"61421d75-0512-490c-ae8e-ed449f7b4ef1\",\r\n \"subActivityId\": \"cddb715f-ae58-4ad3-8057-d1e4ab4131aa\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"af3adfb7-44fa-4141-826e-04cfbf96ab82\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;c2c9a2ce-5b50-4831-9308-2de17897e7e8 ARID=61421d75-0512-490c-ae8e-ed449f7b4ef1 \u003e GW.Http.CallContext/61421d75-0512-490c-ae8e-ed449f7b4ef1 \u003e GW.ExecuteQuery/af3adfb7-44fa-4141-826e-04cfbf96ab82 \u003e DN.FE.ExecuteQuery/cddb715f-ae58-4ad3-8057-d1e4ab4131aa)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:11:57.986]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:11:54.6424073Z);\nlet _endTime = datetime(2026-08-24T21:11:54.6424077Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-nhrxvm5llfcn'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:11:57.8650613Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;c2c9a2ce-5b50-4831-9308-2de17897e7e8\",\r\n \"activityId\": \"61421d75-0512-490c-ae8e-ed449f7b4ef1\",\r\n \"subActivityId\": \"cddb715f-ae58-4ad3-8057-d1e4ab4131aa\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"af3adfb7-44fa-4141-826e-04cfbf96ab82\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;c2c9a2ce-5b50-4831-9308-2de17897e7e8 ARID=61421d75-0512-490c-ae8e-ed449f7b4ef1 \u003e GW.Http.CallContext/61421d75-0512-490c-ae8e-ed449f7b4ef1 \u003e GW.ExecuteQuery/af3adfb7-44fa-4141-826e-04cfbf96ab82 \u003e DN.FE.ExecuteQuery/cddb715f-ae58-4ad3-8057-d1e4ab4131aa)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded 19:54:36 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-fn4s2m" Unexpected error: <fmt.wrapError \| 0xc000dea2e0>: DNS for route host agnhost-e2e-sample-app-jhj46.apps.aro.o4q2a9i7j5z2k6t.i9i4.j4428544.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-jhj46.apps.aro.o4q2a9i7j5z2k6t.i9i4.j4428544.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-jhj46.apps.aro.o4q2a9i7j5z2k6t.i9i4.j4428544.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-fn4s2m" Unexpected error: <fmt.wrapError \| 0xc000dea2e0>: DNS for route host agnhost-e2e-sample-app-jhj46.apps.aro.o4q2a9i7j5z2k6t.i9i4.j4428544.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-jhj46.apps.aro.o4q2a9i7j5z2k6t.i9i4.j4428544.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-jhj46.apps.aro.o4q2a9i7j5z2k6t.i9i4.j4428544.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded ... occurred
19:54:34	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 8ce3ed6	e2e	failed	★#5958 (open)	1	e2e (1) query-basic: command failed: exit status 1 19:54:34 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.008s. Expected success, but got an error: <errors.errorString \| 0xc000b18230>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[21:06:58.789]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:06:55.4059232Z);\nlet _endTime = datetime(2026-08-24T21:06:55.4059238Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-qdhqs95kdb4g'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:06:58.7667615Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;38bc470d-9366-4b94-8899-99c4c881f92e\",\r\n \"activityId\": \"809b48ce-4db2-451c-81c2-e41a8748a2cb\",\r\n \"subActivityId\": \"2b597bb5-a965-4295-9801-672a05444170\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"9730504b-885f-48fc-870b-b1d8ffdc8543\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;38bc470d-9366-4b94-8899-99c4c881f92e ARID=809b48ce-4db2-451c-81c2-e41a8748a2cb \u003e GW.Http.CallContext/809b48ce-4db2-451c-81c2-e41a8748a2cb \u003e GW.ExecuteQuery/9730504b-885f-48fc-870b-b1d8ffdc8543 \u003e DN.FE.ExecuteQuery/2b597bb5-a965-4295-9801-672a05444170)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:06:58.843]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:06:55.4059232Z);\nlet _endTime = datetime(2026-08-24T21:06:55.4059238Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-qdhqs95kdb4g'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:06:58.7667615Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;38bc470d-9366-4b94-8899-99c4c881f92e\",\r\n \"activityId\": \"809b48ce-4db2-451c-81c2-e41a8748a2cb\",\r\n \"subActivityId\": \"2b597bb5-a965-4295-9801-672a05444170\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"9730504b-885f-48fc-870b-b1d8ffdc8543\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;38bc470d-9366-4b94-8899-99c4c881f92e ARID=809b48ce-4db2-451c-81c2-e41a8748a2cb \u003e GW.Http.CallContext/809b48ce-4db2-451c-81c2-e41a8748a2cb \u003e GW.ExecuteQuery/9730504b-885f-48fc-870b-b1d8ffdc8543 \u003e DN.FE.ExecuteQuery/2b597bb5-a965-4295-9801-672a05444170)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[21:07:02.136]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:06:58.9135625Z);\nlet _endTime = datetime(2026-08-24T21:06:58.9135630Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-qdhqs95kdb4g'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:07:02.1107130Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;883d4f16-b2d5-473d-ad61-2e2eaafa2ac2\",\r\n \"activityId\": \"14258bf4-50f1-41c9-b908-681c17389fa4\",\r\n \"subActivityId\": \"da07376e-e1ee-4ad8-95bf-f24b5bb180b7\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"47829b11-3403-462c-83ee-05b021d04b47\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;883d4f16-b2d5-473d-ad61-2e2eaafa2ac2 ARID=14258bf4-50f1-41c9-b908-681c17389fa4 \u003e GW.Http.CallContext/14258bf4-50f1-41c9-b908-681c17389fa4 \u003e GW.ExecuteQuery/47829b11-3403-462c-83ee-05b021d04b47 \u003e DN.FE.ExecuteQuery/da07376e-e1ee-4ad8-95bf-f24b5bb180b7)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:07:02.183]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:06:58.9135625Z);\nlet _endTime = datetime(2026-08-24T21:06:58.9135630Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-qdhqs95kdb4g'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:07:02.1107130Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;883d4f16-b2d5-473d-ad61-2e2eaafa2ac2\",\r\n \"activityId\": \"14258bf4-50f1-41c9-b908-681c17389fa4\",\r\n \"subActivityId\": \"da07376e-e1ee-4ad8-95bf-f24b5bb180b7\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"47829b11-3403-462c-83ee-05b021d04b47\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;883d4f16-b2d5-473d-ad61-2e2eaafa2ac2 ARID=14258bf4-50f1-41c9-b908-681c17389fa4 \u003e GW.Http.CallContext/14258bf4-50f1-41c9-b908-681c17389fa4 \u003e GW.ExecuteQuery/47829b11-3403-462c-83ee-05b021d04b47 \u003e DN.FE.ExecuteQuery/da07376e-e1ee-4ad8-95bf-f24b5bb180b7)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[21:07:10.268]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:07:06.0371001Z);\nlet _endTime = datetime(2026-08-24T21:07:06.0371007Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-qdhqs95kdb4g'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:07:10.2479420Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;d4eb7db5-1698-4436-b648-0abc283404c8\",\r\n \"activityId\": \"428c9294-ade7-4000-8c97-cf84299bfe87\",\r\n \"subActivityId\": \"845f816d-2c80-4bdc-8b63-dfb3d52faa8d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"eaf67d4e-1868-4364-b7bb-9b967c69c037\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;d4eb7db5-1698-4436-b648-0abc283404c8 ARID=428c9294-ade7-4000-8c97-cf84299bfe87 \u003e GW.Http.CallContext/428c9294-ade7-4000-8c97-cf84299bfe87 \u003e GW.ExecuteQuery/eaf67d4e-1868-4364-b7bb-9b967c69c037 \u003e DN.FE.ExecuteQuery/845f816d-2c80-4bdc-8b63-dfb3d52faa8d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:07:10.361]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:07:06.0371001Z);\nlet _endTime = datetime(2026-08-24T21:07:06.0371007Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-qdhqs95kdb4g'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:07:10.2479420Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;d4eb7db5-1698-4436-b648-0abc283404c8\",\r\n \"activityId\": \"428c9294-ade7-4000-8c97-cf84299bfe87\",\r\n \"subActivityId\": \"845f816d-2c80-4bdc-8b63-dfb3d52faa8d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"eaf67d4e-1868-4364-b7bb-9b967c69c037\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;d4eb7db5-1698-4436-b648-0abc283404c8 ARID=428c9294-ade7-4000-8c97-cf84299bfe87 \u003e GW.Http.CallContext/428c9294-ade7-4000-8c97-cf84299bfe87 \u003e GW.ExecuteQuery/eaf67d4e-1868-4364-b7bb-9b967c69c037 \u003e DN.FE.ExecuteQuery/845f816d-2c80-4bdc-8b63-dfb3d52faa8d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.008s. Expected success, but got an error: <errors.errorString \| 0xc000b18230>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[21:06:58.789]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:06:55.4059232Z);\nlet _endTime = datetime(2026-08-24T21:06:55.4059238Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-qdhqs95kdb4g'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:06:58.7667615Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;38bc470d-9366-4b94-8899-99c4c881f92e\",\r\n \"activityId\": \"809b48ce-4db2-451c-81c2-e41a8748a2cb\",\r\n \"subActivityId\": \"2b597bb5-a965-4295-9801-672a05444170\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"9730504b-885f-48fc-870b-b1d8ffdc8543\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;38bc470d-9366-4b94-8899-99c4c881f92e ARID=809b48ce-4db2-451c-81c2-e41a8748a2cb \u003e GW.Http.CallContext/809b48ce-4db2-451c-81c2-e41a8748a2cb \u003e GW.ExecuteQuery/9730504b-885f-48fc-870b-b1d8ffdc8543 \u003e DN.FE.ExecuteQuery/2b597bb5-a965-4295-9801-672a05444170)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:06:58.843]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:06:55.4059232Z);\nlet _endTime = datetime(2026-08-24T21:06:55.4059238Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-qdhqs95kdb4g'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:06:58.7667615Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;38bc470d-9366-4b94-8899-99c4c881f92e\",\r\n \"activityId\": \"809b48ce-4db2-451c-81c2-e41a8748a2cb\",\r\n \"subActivityId\": \"2b597bb5-a965-4295-9801-672a05444170\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"9730504b-885f-48fc-870b-b1d8ffdc8543\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;38bc470d-9366-4b94-8899-99c4c881f92e ARID=809b48ce-4db2-451c-81c2-e41a8748a2cb \u003e GW.Http.CallContext/809b48ce-4db2-451c-81c2-e41a8748a2cb \u003e GW.ExecuteQuery/9730504b-885f-48fc-870b-b1d8ffdc8543 \u003e DN.FE.ExecuteQuery/2b597bb5-a965-4295-9801-672a05444170)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[21:07:02.136]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:06:58.9135625Z);\nlet _endTime = datetime(2026-08-24T21:06:58.9135630Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-qdhqs95kdb4g'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:07:02.1107130Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;883d4f16-b2d5-473d-ad61-2e2eaafa2ac2\",\r\n \"activityId\": \"14258bf4-50f1-41c9-b908-681c17389fa4\",\r\n \"subActivityId\": \"da07376e-e1ee-4ad8-95bf-f24b5bb180b7\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"47829b11-3403-462c-83ee-05b021d04b47\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;883d4f16-b2d5-473d-ad61-2e2eaafa2ac2 ARID=14258bf4-50f1-41c9-b908-681c17389fa4 \u003e GW.Http.CallContext/14258bf4-50f1-41c9-b908-681c17389fa4 \u003e GW.ExecuteQuery/47829b11-3403-462c-83ee-05b021d04b47 \u003e DN.FE.ExecuteQuery/da07376e-e1ee-4ad8-95bf-f24b5bb180b7)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:07:02.183]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:06:58.9135625Z);\nlet _endTime = datetime(2026-08-24T21:06:58.9135630Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-qdhqs95kdb4g'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:07:02.1107130Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;883d4f16-b2d5-473d-ad61-2e2eaafa2ac2\",\r\n \"activityId\": \"14258bf4-50f1-41c9-b908-681c17389fa4\",\r\n \"subActivityId\": \"da07376e-e1ee-4ad8-95bf-f24b5bb180b7\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"47829b11-3403-462c-83ee-05b021d04b47\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;883d4f16-b2d5-473d-ad61-2e2eaafa2ac2 ARID=14258bf4-50f1-41c9-b908-681c17389fa4 \u003e GW.Http.CallContext/14258bf4-50f1-41c9-b908-681c17389fa4 \u003e GW.ExecuteQuery/47829b11-3403-462c-83ee-05b021d04b47 \u003e DN.FE.ExecuteQuery/da07376e-e1ee-4ad8-95bf-f24b5bb180b7)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[21:07:10.268]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:07:06.0371001Z);\nlet _endTime = datetime(2026-08-24T21:07:06.0371007Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-qdhqs95kdb4g'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:07:10.2479420Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;d4eb7db5-1698-4436-b648-0abc283404c8\",\r\n \"activityId\": \"428c9294-ade7-4000-8c97-cf84299bfe87\",\r\n \"subActivityId\": \"845f816d-2c80-4bdc-8b63-dfb3d52faa8d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"eaf67d4e-1868-4364-b7bb-9b967c69c037\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;d4eb7db5-1698-4436-b648-0abc283404c8 ARID=428c9294-ade7-4000-8c97-cf84299bfe87 \u003e GW.Http.CallContext/428c9294-ade7-4000-8c97-cf84299bfe87 \u003e GW.ExecuteQuery/eaf67d4e-1868-4364-b7bb-9b967c69c037 \u003e DN.FE.ExecuteQuery/845f816d-2c80-4bdc-8b63-dfb3d52faa8d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[21:07:10.361]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T21:07:06.0371001Z);\nlet _endTime = datetime(2026-08-24T21:07:06.0371007Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-qdhqs95kdb4g'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T21:07:10.2479420Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;d4eb7db5-1698-4436-b648-0abc283404c8\",\r\n \"activityId\": \"428c9294-ade7-4000-8c97-cf84299bfe87\",\r\n \"subActivityId\": \"845f816d-2c80-4bdc-8b63-dfb3d52faa8d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"eaf67d4e-1868-4364-b7bb-9b967c69c037\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;d4eb7db5-1698-4436-b648-0abc283404c8 ARID=428c9294-ade7-4000-8c97-cf84299bfe87 \u003e GW.Http.CallContext/428c9294-ade7-4000-8c97-cf84299bfe87 \u003e GW.ExecuteQuery/eaf67d4e-1868-4364-b7bb-9b967c69c037 \u003e DN.FE.ExecuteQuery/845f816d-2c80-4bdc-8b63-dfb3d52faa8d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ...
19:06:19	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 515e34b	alert, e2e	failed	★#6583 (open)	3	e2e (2) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 19:06:19 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-q75lrtsn78tc/cluster-candidate-5-0-fph2pn should provision Unexpected error: <fmt.wrapError \| 0xc000da0360>: failed to create HCP cluster cluster-candidate-5-0-fph2pn: failed waiting for cluster="cluster-candidate-5-0-fph2pn" in resourcegroup="rg-candidate-5-0-q75lrtsn78tc" to finish creating: GET https://rp.j4792192.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e34278a0-6b1d-4ae8-a229-75f5d648affe -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e34278a0-6b1d-4ae8-a229-75f5d648affe", "name": "e34278a0-6b1d-4ae8-a229-75f5d648affe", "status": "Failed", "startTime": "2026-08-24T19:54:23.12480854Z", "endTime": "2026-08-24T20:13:28.144152741Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-q75lrtsn78tc/cluster-candidate-5-0-fph2pn should provision Unexpected error: <fmt.wrapError \| 0xc000da0360>: failed to create HCP cluster cluster-candidate-5-0-fph2pn: failed waiting for cluster="cluster-candidate-5-0-fph2pn" in resourcegroup="rg-candidate-5-0-q75lrtsn78tc" to finish creating: GET https://rp.j4792192.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e34278a0-6b1d-4ae8-a229-75f5d648affe -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e34278a0-6b1d-4ae8-a229-75f5d648affe", "name": "e34278a0-6b1d-4ae8-a229-75f5d648affe", "status": "Failed", "startTime": "2026-08-24T19:54:23.12480854Z", "endTime": "2026-08-24T20:13:28.144152741Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred query-basic: command failed: exit status 1 19:06:19 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.002s. Expected success, but got an error: <errors.errorString \| 0xc00068eed0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[20:18:27.943]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:18:22.8770987Z);\nlet _endTime = datetime(2026-08-24T20:18:22.8770991Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-4gcjkhxx9dcw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:18:27.9010465Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;601b8990-928e-4bd5-baaf-10492f42f600\",\r\n \"activityId\": \"8c3b33c8-751b-4382-b6d8-5b5e1fb04797\",\r\n \"subActivityId\": \"1633d9fb-3c87-4c20-8b1a-e67b0bd3dac5\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"b1e0bb17-ad83-46fc-92fc-9d1509f8290e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;601b8990-928e-4bd5-baaf-10492f42f600 ARID=8c3b33c8-751b-4382-b6d8-5b5e1fb04797 \u003e GW.Http.CallContext/8c3b33c8-751b-4382-b6d8-5b5e1fb04797 \u003e GW.ExecuteQuery/b1e0bb17-ad83-46fc-92fc-9d1509f8290e \u003e DN.FE.ExecuteQuery/1633d9fb-3c87-4c20-8b1a-e67b0bd3dac5)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:18:28.002]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:18:22.8770987Z);\nlet _endTime = datetime(2026-08-24T20:18:22.8770991Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-4gcjkhxx9dcw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:18:27.9010465Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;601b8990-928e-4bd5-baaf-10492f42f600\",\r\n \"activityId\": \"8c3b33c8-751b-4382-b6d8-5b5e1fb04797\",\r\n \"subActivityId\": \"1633d9fb-3c87-4c20-8b1a-e67b0bd3dac5\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"b1e0bb17-ad83-46fc-92fc-9d1509f8290e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;601b8990-928e-4bd5-baaf-10492f42f600 ARID=8c3b33c8-751b-4382-b6d8-5b5e1fb04797 \u003e GW.Http.CallContext/8c3b33c8-751b-4382-b6d8-5b5e1fb04797 \u003e GW.ExecuteQuery/b1e0bb17-ad83-46fc-92fc-9d1509f8290e \u003e DN.FE.ExecuteQuery/1633d9fb-3c87-4c20-8b1a-e67b0bd3dac5)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[20:18:32.230]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:18:28.0692542Z);\nlet _endTime = datetime(2026-08-24T20:18:28.0692547Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-4gcjkhxx9dcw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:18:32.2085425Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;02369198-f751-4e34-b6ef-a041051f9254\",\r\n \"activityId\": \"1599c948-9f39-4778-9b37-6affc214343b\",\r\n \"subActivityId\": \"1032076f-4e50-437f-a461-4aee07de64e1\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"b1877adc-b828-4316-8487-63d92a011239\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;02369198-f751-4e34-b6ef-a041051f9254 ARID=1599c948-9f39-4778-9b37-6affc214343b \u003e GW.Http.CallContext/1599c948-9f39-4778-9b37-6affc214343b \u003e GW.ExecuteQuery/b1877adc-b828-4316-8487-63d92a011239 \u003e DN.FE.ExecuteQuery/1032076f-4e50-437f-a461-4aee07de64e1)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:18:32.289]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:18:28.0692542Z);\nlet _endTime = datetime(2026-08-24T20:18:28.0692547Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-4gcjkhxx9dcw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:18:32.2085425Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;02369198-f751-4e34-b6ef-a041051f9254\",\r\n \"activityId\": \"1599c948-9f39-4778-9b37-6affc214343b\",\r\n \"subActivityId\": \"1032076f-4e50-437f-a461-4aee07de64e1\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"b1877adc-b828-4316-8487-63d92a011239\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;02369198-f751-4e34-b6ef-a041051f9254 ARID=1599c948-9f39-4778-9b37-6affc214343b \u003e GW.Http.CallContext/1599c948-9f39-4778-9b37-6affc214343b \u003e GW.ExecuteQuery/b1877adc-b828-4316-8487-63d92a011239 \u003e DN.FE.ExecuteQuery/1032076f-4e50-437f-a461-4aee07de64e1)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[20:18:43.645]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:18:38.7262370Z);\nlet _endTime = datetime(2026-08-24T20:18:38.7262374Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-4gcjkhxx9dcw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:18:43.6200013Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;0e5312a1-dc13-4b0b-b757-efc107708f9a\",\r\n \"activityId\": \"493afc98-c82c-491d-bad6-b6399d34f292\",\r\n \"subActivityId\": \"2090c06b-600c-449f-a94f-d27a4962de6b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"d11d9b18-a372-4c5e-9e48-09784e50e381\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;0e5312a1-dc13-4b0b-b757-efc107708f9a ARID=493afc98-c82c-491d-bad6-b6399d34f292 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/8cc66c09-107e-42be-8f29-290d0f6a372a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/8cc66c09-107e-42be-8f29-290d0f6a372a \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/d11d9b18-a372-4c5e-9e48-09784e50e381 \u003e DN.FE.ExecuteQuery/2090c06b-600c-449f-a94f-d27a4962de6b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:18:43.749]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:18:38.7262370Z);\nlet _endTime = datetime(2026-08-24T20:18:38.7262374Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-4gcjkhxx9dcw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:18:43.6200013Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;0e5312a1-dc13-4b0b-b757-efc107708f9a\",\r\n \"activityId\": \"493afc98-c82c-491d-bad6-b6399d34f292\",\r\n \"subActivityId\": \"2090c06b-600c-449f-a94f-d27a4962de6b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"d11d9b18-a372-4c5e-9e48-09784e50e381\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;0e5312a1-dc13-4b0b-b757-efc107708f9a ARID=493afc98-c82c-491d-bad6-b6399d34f292 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/8cc66c09-107e-42be-8f29-290d0f6a372a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/8cc66c09-107e-42be-8f29-290d0f6a372a \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/d11d9b18-a372-4c5e-9e48-09784e50e381 \u003e DN.FE.ExecuteQuery/2090c06b-600c-449f-a94f-d27a4962de6b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.002s. Expected success, but got an error: <errors.errorString \| 0xc00068eed0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[20:18:27.943]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:18:22.8770987Z);\nlet _endTime = datetime(2026-08-24T20:18:22.8770991Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-4gcjkhxx9dcw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:18:27.9010465Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;601b8990-928e-4bd5-baaf-10492f42f600\",\r\n \"activityId\": \"8c3b33c8-751b-4382-b6d8-5b5e1fb04797\",\r\n \"subActivityId\": \"1633d9fb-3c87-4c20-8b1a-e67b0bd3dac5\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"b1e0bb17-ad83-46fc-92fc-9d1509f8290e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;601b8990-928e-4bd5-baaf-10492f42f600 ARID=8c3b33c8-751b-4382-b6d8-5b5e1fb04797 \u003e GW.Http.CallContext/8c3b33c8-751b-4382-b6d8-5b5e1fb04797 \u003e GW.ExecuteQuery/b1e0bb17-ad83-46fc-92fc-9d1509f8290e \u003e DN.FE.ExecuteQuery/1633d9fb-3c87-4c20-8b1a-e67b0bd3dac5)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:18:28.002]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:18:22.8770987Z);\nlet _endTime = datetime(2026-08-24T20:18:22.8770991Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-4gcjkhxx9dcw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:18:27.9010465Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;601b8990-928e-4bd5-baaf-10492f42f600\",\r\n \"activityId\": \"8c3b33c8-751b-4382-b6d8-5b5e1fb04797\",\r\n \"subActivityId\": \"1633d9fb-3c87-4c20-8b1a-e67b0bd3dac5\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"b1e0bb17-ad83-46fc-92fc-9d1509f8290e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;601b8990-928e-4bd5-baaf-10492f42f600 ARID=8c3b33c8-751b-4382-b6d8-5b5e1fb04797 \u003e GW.Http.CallContext/8c3b33c8-751b-4382-b6d8-5b5e1fb04797 \u003e GW.ExecuteQuery/b1e0bb17-ad83-46fc-92fc-9d1509f8290e \u003e DN.FE.ExecuteQuery/1633d9fb-3c87-4c20-8b1a-e67b0bd3dac5)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[20:18:32.230]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:18:28.0692542Z);\nlet _endTime = datetime(2026-08-24T20:18:28.0692547Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-4gcjkhxx9dcw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:18:32.2085425Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;02369198-f751-4e34-b6ef-a041051f9254\",\r\n \"activityId\": \"1599c948-9f39-4778-9b37-6affc214343b\",\r\n \"subActivityId\": \"1032076f-4e50-437f-a461-4aee07de64e1\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"b1877adc-b828-4316-8487-63d92a011239\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;02369198-f751-4e34-b6ef-a041051f9254 ARID=1599c948-9f39-4778-9b37-6affc214343b \u003e GW.Http.CallContext/1599c948-9f39-4778-9b37-6affc214343b \u003e GW.ExecuteQuery/b1877adc-b828-4316-8487-63d92a011239 \u003e DN.FE.ExecuteQuery/1032076f-4e50-437f-a461-4aee07de64e1)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:18:32.289]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:18:28.0692542Z);\nlet _endTime = datetime(2026-08-24T20:18:28.0692547Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-4gcjkhxx9dcw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:18:32.2085425Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;02369198-f751-4e34-b6ef-a041051f9254\",\r\n \"activityId\": \"1599c948-9f39-4778-9b37-6affc214343b\",\r\n \"subActivityId\": \"1032076f-4e50-437f-a461-4aee07de64e1\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"b1877adc-b828-4316-8487-63d92a011239\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;02369198-f751-4e34-b6ef-a041051f9254 ARID=1599c948-9f39-4778-9b37-6affc214343b \u003e GW.Http.CallContext/1599c948-9f39-4778-9b37-6affc214343b \u003e GW.ExecuteQuery/b1877adc-b828-4316-8487-63d92a011239 \u003e DN.FE.ExecuteQuery/1032076f-4e50-437f-a461-4aee07de64e1)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[20:18:43.645]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:18:38.7262370Z);\nlet _endTime = datetime(2026-08-24T20:18:38.7262374Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-4gcjkhxx9dcw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:18:43.6200013Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;0e5312a1-dc13-4b0b-b757-efc107708f9a\",\r\n \"activityId\": \"493afc98-c82c-491d-bad6-b6399d34f292\",\r\n \"subActivityId\": \"2090c06b-600c-449f-a94f-d27a4962de6b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"d11d9b18-a372-4c5e-9e48-09784e50e381\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;0e5312a1-dc13-4b0b-b757-efc107708f9a ARID=493afc98-c82c-491d-bad6-b6399d34f292 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/8cc66c09-107e-42be-8f29-290d0f6a372a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/8cc66c09-107e-42be-8f29-290d0f6a372a \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/d11d9b18-a372-4c5e-9e48-09784e50e381 \u003e DN.FE.ExecuteQuery/2090c06b-600c-449f-a94f-d27a4962de6b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:18:43.749]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:18:38.7262370Z);\nlet _endTime = datetime(2026-08-24T20:18:38.7262374Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-4gcjkhxx9dcw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:18:43.6200013Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;0e5312a1-dc13-4b0b-b757-efc107708f9a\",\r\n \"activityId\": \"493afc98-c82c-491d-bad6-b6399d34f292\",\r\n \"subActivityId\": \"2090c06b-600c-449f-a94f-d27a4962de6b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"d11d9b18-a372-4c5e-9e48-09784e50e381\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;0e5312a1-dc13-4b0b-b757-efc107708f9a ARID=493afc98-c82c-491d-bad6-b6399d34f292 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/8cc66c09-107e-42be-8f29-290d0f6a372a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/8cc66c09-107e-42be-8f29-290d0f6a372a \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/d11d9b18-a372-4c5e-9e48-09784e50e381 \u003e DN.FE.ExecuteQuery/2090c06b-600c-449f-a94f-d27a4962de6b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... alert (1) alert [hcp] KubePodNotReady fired 19:06:19 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-24T20:07:35Z Ended: 2026-08-24T20:22:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j4792192-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd56rvedr5tvpk5tgrh8ec0h63s5l3e-m5s0s1f2s8t9h9z", pod="router-698586c75d-g7ftd", severity="warning" Description: Pod ocm-arohcpci01-2sd56rvedr5tvpk5tgrh8ec0h63s5l3e-m5s0s1f2s8t9h9z/router-698586c75d-g7ftd has been in a non-ready state for longer than 5 minutes.
18:58:00	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c403afe	e2e	failed	★#6638 (open)	2	e2e (2) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 18:58:00 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--6mjzd2t8mqtv/cluster-hs-pre-wv58r8 should provision Unexpected error: <fmt.wrapError \| 0xc000668260>: failed to create HCP cluster cluster-hs-pre-wv58r8: failed waiting for cluster="cluster-hs-pre-wv58r8" in resourcegroup="rg-hs-pre--6mjzd2t8mqtv" to finish creating: GET https://rp.j4302720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5eab453e-b10a-4490-8636-54b57901a948 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5eab453e-b10a-4490-8636-54b57901a948", "name": "5eab453e-b10a-4490-8636-54b57901a948", "status": "Failed", "startTime": "2026-08-24T19:44:35.517103456Z", "endTime": "2026-08-24T20:03:43.794426957Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--6mjzd2t8mqtv/cluster-hs-pre-wv58r8 should provision Unexpected error: <fmt.wrapError \| 0xc000668260>: failed to create HCP cluster cluster-hs-pre-wv58r8: failed waiting for cluster="cluster-hs-pre-wv58r8" in resourcegroup="rg-hs-pre--6mjzd2t8mqtv" to finish creating: GET https://rp.j4302720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5eab453e-b10a-4490-8636-54b57901a948 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5eab453e-b10a-4490-8636-54b57901a948", "name": "5eab453e-b10a-4490-8636-54b57901a948", "status": "Failed", "startTime": "2026-08-24T19:44:35.517103456Z", "endTime": "2026-08-24T20:03:43.794426957Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred query-basic: command failed: exit status 1 18:58:00 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.007s. Expected success, but got an error: <errors.errorString \| 0xc00084e500>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[20:11:06.365]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:11:03.1104533Z);\nlet _endTime = datetime(2026-08-24T20:11:03.1104538Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xnm2zp8mkw5p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:11:06.3437566Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;b4cbf958-ca9a-43ef-b045-0cee872da282\",\r\n \"activityId\": \"cfd929d4-6a8a-483d-8797-30fd98f59ea0\",\r\n \"subActivityId\": \"70f461c7-3c0c-41c5-931e-7af17d9a102d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f45a7835-de0e-40e8-9b65-69f97a873ce5\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;b4cbf958-ca9a-43ef-b045-0cee872da282 ARID=cfd929d4-6a8a-483d-8797-30fd98f59ea0 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/be02d506-c5a8-4381-824a-4e22e72b0fa9 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/be02d506-c5a8-4381-824a-4e22e72b0fa9 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/f45a7835-de0e-40e8-9b65-69f97a873ce5 \u003e DN.FE.ExecuteQuery/70f461c7-3c0c-41c5-931e-7af17d9a102d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:11:06.417]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:11:03.1104533Z);\nlet _endTime = datetime(2026-08-24T20:11:03.1104538Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xnm2zp8mkw5p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:11:06.3437566Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;b4cbf958-ca9a-43ef-b045-0cee872da282\",\r\n \"activityId\": \"cfd929d4-6a8a-483d-8797-30fd98f59ea0\",\r\n \"subActivityId\": \"70f461c7-3c0c-41c5-931e-7af17d9a102d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f45a7835-de0e-40e8-9b65-69f97a873ce5\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;b4cbf958-ca9a-43ef-b045-0cee872da282 ARID=cfd929d4-6a8a-483d-8797-30fd98f59ea0 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/be02d506-c5a8-4381-824a-4e22e72b0fa9 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/be02d506-c5a8-4381-824a-4e22e72b0fa9 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/f45a7835-de0e-40e8-9b65-69f97a873ce5 \u003e DN.FE.ExecuteQuery/70f461c7-3c0c-41c5-931e-7af17d9a102d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[20:11:09.781]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:11:06.4896507Z);\nlet _endTime = datetime(2026-08-24T20:11:06.4896513Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xnm2zp8mkw5p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:11:09.7597259Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;b7256059-edf2-4c2f-80f2-3bee9c0305c7\",\r\n \"activityId\": \"8d86dded-b5f8-4d3b-9e9a-8b2f53a3c3ab\",\r\n \"subActivityId\": \"ea2d196d-0243-4e19-91e6-6f5f837fa783\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"34771c8b-9434-4012-a5f3-a7296d99e393\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;b7256059-edf2-4c2f-80f2-3bee9c0305c7 ARID=8d86dded-b5f8-4d3b-9e9a-8b2f53a3c3ab \u003e GW.Http.CallContext/8d86dded-b5f8-4d3b-9e9a-8b2f53a3c3ab \u003e GW.ExecuteQuery/34771c8b-9434-4012-a5f3-a7296d99e393 \u003e DN.FE.ExecuteQuery/ea2d196d-0243-4e19-91e6-6f5f837fa783)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:11:09.842]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:11:06.4896507Z);\nlet _endTime = datetime(2026-08-24T20:11:06.4896513Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xnm2zp8mkw5p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:11:09.7597259Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;b7256059-edf2-4c2f-80f2-3bee9c0305c7\",\r\n \"activityId\": \"8d86dded-b5f8-4d3b-9e9a-8b2f53a3c3ab\",\r\n \"subActivityId\": \"ea2d196d-0243-4e19-91e6-6f5f837fa783\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"34771c8b-9434-4012-a5f3-a7296d99e393\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;b7256059-edf2-4c2f-80f2-3bee9c0305c7 ARID=8d86dded-b5f8-4d3b-9e9a-8b2f53a3c3ab \u003e GW.Http.CallContext/8d86dded-b5f8-4d3b-9e9a-8b2f53a3c3ab \u003e GW.ExecuteQuery/34771c8b-9434-4012-a5f3-a7296d99e393 \u003e DN.FE.ExecuteQuery/ea2d196d-0243-4e19-91e6-6f5f837fa783)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[20:11:16.142]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:11:13.0624028Z);\nlet _endTime = datetime(2026-08-24T20:11:13.0624037Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xnm2zp8mkw5p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:11:16.1213348Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;efeb4b63-c2c4-476e-8686-4e74080c884a\",\r\n \"activityId\": \"382114ad-bdad-42eb-b0fe-95ae67f19abb\",\r\n \"subActivityId\": \"7bae5a33-a42c-4b39-bcd3-5b037a769b51\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"6f208b47-b7ff-40cd-9704-f6cfae775429\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;efeb4b63-c2c4-476e-8686-4e74080c884a ARID=382114ad-bdad-42eb-b0fe-95ae67f19abb \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/c2626d9e-6d97-4479-897b-c39ca1bcc4f6 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/c2626d9e-6d97-4479-897b-c39ca1bcc4f6 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/6f208b47-b7ff-40cd-9704-f6cfae775429 \u003e DN.FE.ExecuteQuery/7bae5a33-a42c-4b39-bcd3-5b037a769b51)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:11:16.256]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:11:13.0624028Z);\nlet _endTime = datetime(2026-08-24T20:11:13.0624037Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xnm2zp8mkw5p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:11:16.1213348Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;efeb4b63-c2c4-476e-8686-4e74080c884a\",\r\n \"activityId\": \"382114ad-bdad-42eb-b0fe-95ae67f19abb\",\r\n \"subActivityId\": \"7bae5a33-a42c-4b39-bcd3-5b037a769b51\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"6f208b47-b7ff-40cd-9704-f6cfae775429\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;efeb4b63-c2c4-476e-8686-4e74080c884a ARID=382114ad-bdad-42eb-b0fe-95ae67f19abb \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/c2626d9e-6d97-4479-897b-c39ca1bcc4f6 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/c2626d9e-6d97-4479-897b-c39ca1bcc4f6 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/6f208b47-b7ff-40cd-9704-f6cfae775429 \u003e DN.FE.ExecuteQuery/7bae5a33-a42c-4b39-bcd3-5b037a769b51)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.007s. Expected success, but got an error: <errors.errorString \| 0xc00084e500>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[20:11:06.365]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:11:03.1104533Z);\nlet _endTime = datetime(2026-08-24T20:11:03.1104538Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xnm2zp8mkw5p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:11:06.3437566Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;b4cbf958-ca9a-43ef-b045-0cee872da282\",\r\n \"activityId\": \"cfd929d4-6a8a-483d-8797-30fd98f59ea0\",\r\n \"subActivityId\": \"70f461c7-3c0c-41c5-931e-7af17d9a102d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f45a7835-de0e-40e8-9b65-69f97a873ce5\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;b4cbf958-ca9a-43ef-b045-0cee872da282 ARID=cfd929d4-6a8a-483d-8797-30fd98f59ea0 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/be02d506-c5a8-4381-824a-4e22e72b0fa9 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/be02d506-c5a8-4381-824a-4e22e72b0fa9 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/f45a7835-de0e-40e8-9b65-69f97a873ce5 \u003e DN.FE.ExecuteQuery/70f461c7-3c0c-41c5-931e-7af17d9a102d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:11:06.417]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:11:03.1104533Z);\nlet _endTime = datetime(2026-08-24T20:11:03.1104538Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xnm2zp8mkw5p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:11:06.3437566Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;b4cbf958-ca9a-43ef-b045-0cee872da282\",\r\n \"activityId\": \"cfd929d4-6a8a-483d-8797-30fd98f59ea0\",\r\n \"subActivityId\": \"70f461c7-3c0c-41c5-931e-7af17d9a102d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f45a7835-de0e-40e8-9b65-69f97a873ce5\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;b4cbf958-ca9a-43ef-b045-0cee872da282 ARID=cfd929d4-6a8a-483d-8797-30fd98f59ea0 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/be02d506-c5a8-4381-824a-4e22e72b0fa9 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/be02d506-c5a8-4381-824a-4e22e72b0fa9 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/f45a7835-de0e-40e8-9b65-69f97a873ce5 \u003e DN.FE.ExecuteQuery/70f461c7-3c0c-41c5-931e-7af17d9a102d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[20:11:09.781]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:11:06.4896507Z);\nlet _endTime = datetime(2026-08-24T20:11:06.4896513Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xnm2zp8mkw5p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:11:09.7597259Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;b7256059-edf2-4c2f-80f2-3bee9c0305c7\",\r\n \"activityId\": \"8d86dded-b5f8-4d3b-9e9a-8b2f53a3c3ab\",\r\n \"subActivityId\": \"ea2d196d-0243-4e19-91e6-6f5f837fa783\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"34771c8b-9434-4012-a5f3-a7296d99e393\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;b7256059-edf2-4c2f-80f2-3bee9c0305c7 ARID=8d86dded-b5f8-4d3b-9e9a-8b2f53a3c3ab \u003e GW.Http.CallContext/8d86dded-b5f8-4d3b-9e9a-8b2f53a3c3ab \u003e GW.ExecuteQuery/34771c8b-9434-4012-a5f3-a7296d99e393 \u003e DN.FE.ExecuteQuery/ea2d196d-0243-4e19-91e6-6f5f837fa783)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:11:09.842]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:11:06.4896507Z);\nlet _endTime = datetime(2026-08-24T20:11:06.4896513Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xnm2zp8mkw5p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:11:09.7597259Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;b7256059-edf2-4c2f-80f2-3bee9c0305c7\",\r\n \"activityId\": \"8d86dded-b5f8-4d3b-9e9a-8b2f53a3c3ab\",\r\n \"subActivityId\": \"ea2d196d-0243-4e19-91e6-6f5f837fa783\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"34771c8b-9434-4012-a5f3-a7296d99e393\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;b7256059-edf2-4c2f-80f2-3bee9c0305c7 ARID=8d86dded-b5f8-4d3b-9e9a-8b2f53a3c3ab \u003e GW.Http.CallContext/8d86dded-b5f8-4d3b-9e9a-8b2f53a3c3ab \u003e GW.ExecuteQuery/34771c8b-9434-4012-a5f3-a7296d99e393 \u003e DN.FE.ExecuteQuery/ea2d196d-0243-4e19-91e6-6f5f837fa783)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[20:11:16.142]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:11:13.0624028Z);\nlet _endTime = datetime(2026-08-24T20:11:13.0624037Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xnm2zp8mkw5p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:11:16.1213348Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;efeb4b63-c2c4-476e-8686-4e74080c884a\",\r\n \"activityId\": \"382114ad-bdad-42eb-b0fe-95ae67f19abb\",\r\n \"subActivityId\": \"7bae5a33-a42c-4b39-bcd3-5b037a769b51\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"6f208b47-b7ff-40cd-9704-f6cfae775429\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;efeb4b63-c2c4-476e-8686-4e74080c884a ARID=382114ad-bdad-42eb-b0fe-95ae67f19abb \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/c2626d9e-6d97-4479-897b-c39ca1bcc4f6 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/c2626d9e-6d97-4479-897b-c39ca1bcc4f6 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/6f208b47-b7ff-40cd-9704-f6cfae775429 \u003e DN.FE.ExecuteQuery/7bae5a33-a42c-4b39-bcd3-5b037a769b51)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:11:16.256]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:11:13.0624028Z);\nlet _endTime = datetime(2026-08-24T20:11:13.0624037Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xnm2zp8mkw5p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:11:16.1213348Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;efeb4b63-c2c4-476e-8686-4e74080c884a\",\r\n \"activityId\": \"382114ad-bdad-42eb-b0fe-95ae67f19abb\",\r\n \"subActivityId\": \"7bae5a33-a42c-4b39-bcd3-5b037a769b51\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"6f208b47-b7ff-40cd-9704-f6cfae775429\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;efeb4b63-c2c4-476e-8686-4e74080c884a ARID=382114ad-bdad-42eb-b0fe-95ae67f19abb \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/c2626d9e-6d97-4479-897b-c39ca1bcc4f6 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/c2626d9e-6d97-4479-897b-c39ca1bcc4f6 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/6f208b47-b7ff-40cd-9704-f6cfae775429 \u003e DN.FE.ExecuteQuery/7bae5a33-a42c-4b39-bcd3-5b037a769b51)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ...
18:57:56	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 54bb2db	unknown	failed	#6575 (open)	1	non-artifact-backed failure (no junit artifacts)
18:57:49	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c9b8452	e2e	failed	★#6584 (open)	1	e2e (1) query-basic: command failed: exit status 1 18:57:49 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 601.015s. Expected success, but got an error: <errors.errorString \| 0xc0007f4420>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[20:27:30.508]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:27:26.9752930Z);\nlet _endTime = datetime(2026-08-24T20:27:26.9752934Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-7cvwsv4nwzpc'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:27:30.4873262Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;79d3ac0e-11fc-415c-b3f4-7428bf5d3dd3\",\r\n \"activityId\": \"8b77f6d0-685b-4580-b228-7fef995a3082\",\r\n \"subActivityId\": \"dde5fd86-b4e1-4ab1-9e45-83f9419538b9\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"feade8fe-ce29-4ae5-8877-792ab307c6cd\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;79d3ac0e-11fc-415c-b3f4-7428bf5d3dd3 ARID=8b77f6d0-685b-4580-b228-7fef995a3082 \u003e GW.Http.CallContext/8b77f6d0-685b-4580-b228-7fef995a3082 \u003e GW.ExecuteQuery/feade8fe-ce29-4ae5-8877-792ab307c6cd \u003e DN.FE.ExecuteQuery/dde5fd86-b4e1-4ab1-9e45-83f9419538b9)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:27:30.563]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:27:26.9752930Z);\nlet _endTime = datetime(2026-08-24T20:27:26.9752934Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-7cvwsv4nwzpc'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:27:30.4873262Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;79d3ac0e-11fc-415c-b3f4-7428bf5d3dd3\",\r\n \"activityId\": \"8b77f6d0-685b-4580-b228-7fef995a3082\",\r\n \"subActivityId\": \"dde5fd86-b4e1-4ab1-9e45-83f9419538b9\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"feade8fe-ce29-4ae5-8877-792ab307c6cd\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;79d3ac0e-11fc-415c-b3f4-7428bf5d3dd3 ARID=8b77f6d0-685b-4580-b228-7fef995a3082 \u003e GW.Http.CallContext/8b77f6d0-685b-4580-b228-7fef995a3082 \u003e GW.ExecuteQuery/feade8fe-ce29-4ae5-8877-792ab307c6cd \u003e DN.FE.ExecuteQuery/dde5fd86-b4e1-4ab1-9e45-83f9419538b9)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[20:27:33.858]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:27:30.6324586Z);\nlet _endTime = datetime(2026-08-24T20:27:30.6324590Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-7cvwsv4nwzpc'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:27:33.8337270Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;c03b81b7-835a-4204-b9d7-986eb63c212e\",\r\n \"activityId\": \"72d04e18-99e2-42a4-9077-70feb0186d65\",\r\n \"subActivityId\": \"3bed1316-100e-49f0-8b79-112dca484d33\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"2930eef7-01a7-4653-85aa-a04fd7b6171e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;c03b81b7-835a-4204-b9d7-986eb63c212e ARID=72d04e18-99e2-42a4-9077-70feb0186d65 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/1c0446ed-dd2a-4622-a43c-42d5895105ed \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/1c0446ed-dd2a-4622-a43c-42d5895105ed \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/2930eef7-01a7-4653-85aa-a04fd7b6171e \u003e DN.FE.ExecuteQuery/3bed1316-100e-49f0-8b79-112dca484d33)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:27:33.914]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:27:30.6324586Z);\nlet _endTime = datetime(2026-08-24T20:27:30.6324590Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-7cvwsv4nwzpc'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:27:33.8337270Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;c03b81b7-835a-4204-b9d7-986eb63c212e\",\r\n \"activityId\": \"72d04e18-99e2-42a4-9077-70feb0186d65\",\r\n \"subActivityId\": \"3bed1316-100e-49f0-8b79-112dca484d33\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"2930eef7-01a7-4653-85aa-a04fd7b6171e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;c03b81b7-835a-4204-b9d7-986eb63c212e ARID=72d04e18-99e2-42a4-9077-70feb0186d65 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/1c0446ed-dd2a-4622-a43c-42d5895105ed \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/1c0446ed-dd2a-4622-a43c-42d5895105ed \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/2930eef7-01a7-4653-85aa-a04fd7b6171e \u003e DN.FE.ExecuteQuery/3bed1316-100e-49f0-8b79-112dca484d33)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[20:27:40.748]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:27:37.4605711Z);\nlet _endTime = datetime(2026-08-24T20:27:37.4605715Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-7cvwsv4nwzpc'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:27:40.7261539Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;f2096b01-fc59-4a87-a98c-750912083a20\",\r\n \"activityId\": \"834ce702-d6f1-43ba-8ec3-287e74566d46\",\r\n \"subActivityId\": \"15c9f489-b0a4-4599-85d4-b7bf5e4ce646\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"c5cbfae2-c764-4c2d-a189-4e84f1cad88a\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;f2096b01-fc59-4a87-a98c-750912083a20 ARID=834ce702-d6f1-43ba-8ec3-287e74566d46 \u003e GW.Http.CallContext/834ce702-d6f1-43ba-8ec3-287e74566d46 \u003e GW.ExecuteQuery/c5cbfae2-c764-4c2d-a189-4e84f1cad88a \u003e DN.FE.ExecuteQuery/15c9f489-b0a4-4599-85d4-b7bf5e4ce646)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:27:40.846]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:27:37.4605711Z);\nlet _endTime = datetime(2026-08-24T20:27:37.4605715Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-7cvwsv4nwzpc'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:27:40.7261539Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;f2096b01-fc59-4a87-a98c-750912083a20\",\r\n \"activityId\": \"834ce702-d6f1-43ba-8ec3-287e74566d46\",\r\n \"subActivityId\": \"15c9f489-b0a4-4599-85d4-b7bf5e4ce646\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"c5cbfae2-c764-4c2d-a189-4e84f1cad88a\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;f2096b01-fc59-4a87-a98c-750912083a20 ARID=834ce702-d6f1-43ba-8ec3-287e74566d46 \u003e GW.Http.CallContext/834ce702-d6f1-43ba-8ec3-287e74566d46 \u003e GW.ExecuteQuery/c5cbfae2-c764-4c2d-a189-4e84f1cad88a \u003e DN.FE.ExecuteQuery/15c9f489-b0a4-4599-85d4-b7bf5e4ce646)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 601.015s. Expected success, but got an error: <errors.errorString \| 0xc0007f4420>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[20:27:30.508]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:27:26.9752930Z);\nlet _endTime = datetime(2026-08-24T20:27:26.9752934Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-7cvwsv4nwzpc'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:27:30.4873262Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;79d3ac0e-11fc-415c-b3f4-7428bf5d3dd3\",\r\n \"activityId\": \"8b77f6d0-685b-4580-b228-7fef995a3082\",\r\n \"subActivityId\": \"dde5fd86-b4e1-4ab1-9e45-83f9419538b9\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"feade8fe-ce29-4ae5-8877-792ab307c6cd\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;79d3ac0e-11fc-415c-b3f4-7428bf5d3dd3 ARID=8b77f6d0-685b-4580-b228-7fef995a3082 \u003e GW.Http.CallContext/8b77f6d0-685b-4580-b228-7fef995a3082 \u003e GW.ExecuteQuery/feade8fe-ce29-4ae5-8877-792ab307c6cd \u003e DN.FE.ExecuteQuery/dde5fd86-b4e1-4ab1-9e45-83f9419538b9)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:27:30.563]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:27:26.9752930Z);\nlet _endTime = datetime(2026-08-24T20:27:26.9752934Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-7cvwsv4nwzpc'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:27:30.4873262Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;79d3ac0e-11fc-415c-b3f4-7428bf5d3dd3\",\r\n \"activityId\": \"8b77f6d0-685b-4580-b228-7fef995a3082\",\r\n \"subActivityId\": \"dde5fd86-b4e1-4ab1-9e45-83f9419538b9\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"feade8fe-ce29-4ae5-8877-792ab307c6cd\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;79d3ac0e-11fc-415c-b3f4-7428bf5d3dd3 ARID=8b77f6d0-685b-4580-b228-7fef995a3082 \u003e GW.Http.CallContext/8b77f6d0-685b-4580-b228-7fef995a3082 \u003e GW.ExecuteQuery/feade8fe-ce29-4ae5-8877-792ab307c6cd \u003e DN.FE.ExecuteQuery/dde5fd86-b4e1-4ab1-9e45-83f9419538b9)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[20:27:33.858]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:27:30.6324586Z);\nlet _endTime = datetime(2026-08-24T20:27:30.6324590Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-7cvwsv4nwzpc'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:27:33.8337270Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;c03b81b7-835a-4204-b9d7-986eb63c212e\",\r\n \"activityId\": \"72d04e18-99e2-42a4-9077-70feb0186d65\",\r\n \"subActivityId\": \"3bed1316-100e-49f0-8b79-112dca484d33\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"2930eef7-01a7-4653-85aa-a04fd7b6171e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;c03b81b7-835a-4204-b9d7-986eb63c212e ARID=72d04e18-99e2-42a4-9077-70feb0186d65 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/1c0446ed-dd2a-4622-a43c-42d5895105ed \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/1c0446ed-dd2a-4622-a43c-42d5895105ed \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/2930eef7-01a7-4653-85aa-a04fd7b6171e \u003e DN.FE.ExecuteQuery/3bed1316-100e-49f0-8b79-112dca484d33)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:27:33.914]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:27:30.6324586Z);\nlet _endTime = datetime(2026-08-24T20:27:30.6324590Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-7cvwsv4nwzpc'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:27:33.8337270Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;c03b81b7-835a-4204-b9d7-986eb63c212e\",\r\n \"activityId\": \"72d04e18-99e2-42a4-9077-70feb0186d65\",\r\n \"subActivityId\": \"3bed1316-100e-49f0-8b79-112dca484d33\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"2930eef7-01a7-4653-85aa-a04fd7b6171e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;c03b81b7-835a-4204-b9d7-986eb63c212e ARID=72d04e18-99e2-42a4-9077-70feb0186d65 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/1c0446ed-dd2a-4622-a43c-42d5895105ed \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/1c0446ed-dd2a-4622-a43c-42d5895105ed \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/2930eef7-01a7-4653-85aa-a04fd7b6171e \u003e DN.FE.ExecuteQuery/3bed1316-100e-49f0-8b79-112dca484d33)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[20:27:40.748]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:27:37.4605711Z);\nlet _endTime = datetime(2026-08-24T20:27:37.4605715Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-7cvwsv4nwzpc'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:27:40.7261539Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;f2096b01-fc59-4a87-a98c-750912083a20\",\r\n \"activityId\": \"834ce702-d6f1-43ba-8ec3-287e74566d46\",\r\n \"subActivityId\": \"15c9f489-b0a4-4599-85d4-b7bf5e4ce646\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"c5cbfae2-c764-4c2d-a189-4e84f1cad88a\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;f2096b01-fc59-4a87-a98c-750912083a20 ARID=834ce702-d6f1-43ba-8ec3-287e74566d46 \u003e GW.Http.CallContext/834ce702-d6f1-43ba-8ec3-287e74566d46 \u003e GW.ExecuteQuery/c5cbfae2-c764-4c2d-a189-4e84f1cad88a \u003e DN.FE.ExecuteQuery/15c9f489-b0a4-4599-85d4-b7bf5e4ce646)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:27:40.846]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:27:37.4605711Z);\nlet _endTime = datetime(2026-08-24T20:27:37.4605715Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-7cvwsv4nwzpc'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:27:40.7261539Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;f2096b01-fc59-4a87-a98c-750912083a20\",\r\n \"activityId\": \"834ce702-d6f1-43ba-8ec3-287e74566d46\",\r\n \"subActivityId\": \"15c9f489-b0a4-4599-85d4-b7bf5e4ce646\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"c5cbfae2-c764-4c2d-a189-4e84f1cad88a\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;f2096b01-fc59-4a87-a98c-750912083a20 ARID=834ce702-d6f1-43ba-8ec3-287e74566d46 \u003e GW.Http.CallContext/834ce702-d6f1-43ba-8ec3-287e74566d46 \u003e GW.ExecuteQuery/c5cbfae2-c764-4c2d-a189-4e84f1cad88a \u003e DN.FE.ExecuteQuery/15c9f489-b0a4-4599-85d4-b7bf5e4ce646)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ...
18:53:38	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 510b664	alert, e2e	failed	★#6648 (open)	5	e2e (3) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 18:53:38 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc000eacb00>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-9vb844mdss2z" to finish creating: GET https://rp.j4068096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/950ebef7-ebd5-431d-8b1a-af617af56c1e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/950ebef7-ebd5-431d-8b1a-af617af56c1e", "name": "950ebef7-ebd5-431d-8b1a-af617af56c1e", "status": "Failed", "startTime": "2026-08-24T19:41:55.60628386Z", "endTime": "2026-08-24T20:01:00.110147026Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc000eacb00>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-9vb844mdss2z" to finish creating: GET https://rp.j4068096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/950ebef7-ebd5-431d-8b1a-af617af56c1e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/950ebef7-ebd5-431d-8b1a-af617af56c1e", "name": "950ebef7-ebd5-431d-8b1a-af617af56c1e", "status": "Failed", "startTime": "2026-08-24T19:41:55.60628386Z", "endTime": "2026-08-24T20:01:00.110147026Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred query-basic: command failed: exit status 1 18:53:38 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.007s. Expected success, but got an error: <errors.errorString \| 0xc001978690>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[20:06:16.734]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:06:13.3553172Z);\nlet _endTime = datetime(2026-08-24T20:06:13.3553175Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-9p7d8t7v9xz6'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:06:16.7133479Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;34da49f1-b8c4-423d-9a0d-9594ca2081bc\",\r\n \"activityId\": \"2f032998-f59c-4ae2-a694-38cc61f6ea94\",\r\n \"subActivityId\": \"554417e1-fc52-46f3-b197-f82713ea9e34\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"7e75cd47-6911-43a3-a862-d633327c1574\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;34da49f1-b8c4-423d-9a0d-9594ca2081bc ARID=2f032998-f59c-4ae2-a694-38cc61f6ea94 \u003e GW.Http.CallContext/2f032998-f59c-4ae2-a694-38cc61f6ea94 \u003e GW.ExecuteQuery/7e75cd47-6911-43a3-a862-d633327c1574 \u003e DN.FE.ExecuteQuery/554417e1-fc52-46f3-b197-f82713ea9e34)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:06:16.786]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:06:13.3553172Z);\nlet _endTime = datetime(2026-08-24T20:06:13.3553175Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-9p7d8t7v9xz6'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:06:16.7133479Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;34da49f1-b8c4-423d-9a0d-9594ca2081bc\",\r\n \"activityId\": \"2f032998-f59c-4ae2-a694-38cc61f6ea94\",\r\n \"subActivityId\": \"554417e1-fc52-46f3-b197-f82713ea9e34\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"7e75cd47-6911-43a3-a862-d633327c1574\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;34da49f1-b8c4-423d-9a0d-9594ca2081bc ARID=2f032998-f59c-4ae2-a694-38cc61f6ea94 \u003e GW.Http.CallContext/2f032998-f59c-4ae2-a694-38cc61f6ea94 \u003e GW.ExecuteQuery/7e75cd47-6911-43a3-a862-d633327c1574 \u003e DN.FE.ExecuteQuery/554417e1-fc52-46f3-b197-f82713ea9e34)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[20:06:20.207]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:06:16.8552371Z);\nlet _endTime = datetime(2026-08-24T20:06:16.8552374Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-9p7d8t7v9xz6'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:06:20.1847522Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;8f097191-494f-4ed8-bac2-68eee8e5ff98\",\r\n \"activityId\": \"f091c01f-7b20-44b8-b04f-fb0467df4e03\",\r\n \"subActivityId\": \"7026bbd3-2c23-4814-a331-0658c9a9b209\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"c2506c2f-7658-4205-bf77-f1c430893684\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;8f097191-494f-4ed8-bac2-68eee8e5ff98 ARID=f091c01f-7b20-44b8-b04f-fb0467df4e03 \u003e GW.Http.CallContext/f091c01f-7b20-44b8-b04f-fb0467df4e03 \u003e GW.ExecuteQuery/c2506c2f-7658-4205-bf77-f1c430893684 \u003e DN.FE.ExecuteQuery/7026bbd3-2c23-4814-a331-0658c9a9b209)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:06:20.258]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:06:16.8552371Z);\nlet _endTime = datetime(2026-08-24T20:06:16.8552374Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-9p7d8t7v9xz6'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:06:20.1847522Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;8f097191-494f-4ed8-bac2-68eee8e5ff98\",\r\n \"activityId\": \"f091c01f-7b20-44b8-b04f-fb0467df4e03\",\r\n \"subActivityId\": \"7026bbd3-2c23-4814-a331-0658c9a9b209\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"c2506c2f-7658-4205-bf77-f1c430893684\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;8f097191-494f-4ed8-bac2-68eee8e5ff98 ARID=f091c01f-7b20-44b8-b04f-fb0467df4e03 \u003e GW.Http.CallContext/f091c01f-7b20-44b8-b04f-fb0467df4e03 \u003e GW.ExecuteQuery/c2506c2f-7658-4205-bf77-f1c430893684 \u003e DN.FE.ExecuteQuery/7026bbd3-2c23-4814-a331-0658c9a9b209)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[20:06:27.216]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:06:23.8328269Z);\nlet _endTime = datetime(2026-08-24T20:06:23.8328273Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-9p7d8t7v9xz6'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:06:27.1947859Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;e15ec81f-eef5-49f2-8e23-4fd3c09c50c6\",\r\n \"activityId\": \"f3d06959-0b4e-4ef2-9dda-5ba6f765681d\",\r\n \"subActivityId\": \"31d47128-d285-43b5-80f5-22705e153fd2\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"782870fe-7874-4d64-93ab-867f2156413c\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;e15ec81f-eef5-49f2-8e23-4fd3c09c50c6 ARID=f3d06959-0b4e-4ef2-9dda-5ba6f765681d \u003e GW.Http.CallContext/f3d06959-0b4e-4ef2-9dda-5ba6f765681d \u003e GW.ExecuteQuery/782870fe-7874-4d64-93ab-867f2156413c \u003e DN.FE.ExecuteQuery/31d47128-d285-43b5-80f5-22705e153fd2)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:06:27.336]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:06:23.8328269Z);\nlet _endTime = datetime(2026-08-24T20:06:23.8328273Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-9p7d8t7v9xz6'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:06:27.1947859Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;e15ec81f-eef5-49f2-8e23-4fd3c09c50c6\",\r\n \"activityId\": \"f3d06959-0b4e-4ef2-9dda-5ba6f765681d\",\r\n \"subActivityId\": \"31d47128-d285-43b5-80f5-22705e153fd2\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"782870fe-7874-4d64-93ab-867f2156413c\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;e15ec81f-eef5-49f2-8e23-4fd3c09c50c6 ARID=f3d06959-0b4e-4ef2-9dda-5ba6f765681d \u003e GW.Http.CallContext/f3d06959-0b4e-4ef2-9dda-5ba6f765681d \u003e GW.ExecuteQuery/782870fe-7874-4d64-93ab-867f2156413c \u003e DN.FE.ExecuteQuery/31d47128-d285-43b5-80f5-22705e153fd2)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.007s. Expected success, but got an error: <errors.errorString \| 0xc001978690>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[20:06:16.734]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:06:13.3553172Z);\nlet _endTime = datetime(2026-08-24T20:06:13.3553175Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-9p7d8t7v9xz6'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:06:16.7133479Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;34da49f1-b8c4-423d-9a0d-9594ca2081bc\",\r\n \"activityId\": \"2f032998-f59c-4ae2-a694-38cc61f6ea94\",\r\n \"subActivityId\": \"554417e1-fc52-46f3-b197-f82713ea9e34\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"7e75cd47-6911-43a3-a862-d633327c1574\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;34da49f1-b8c4-423d-9a0d-9594ca2081bc ARID=2f032998-f59c-4ae2-a694-38cc61f6ea94 \u003e GW.Http.CallContext/2f032998-f59c-4ae2-a694-38cc61f6ea94 \u003e GW.ExecuteQuery/7e75cd47-6911-43a3-a862-d633327c1574 \u003e DN.FE.ExecuteQuery/554417e1-fc52-46f3-b197-f82713ea9e34)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:06:16.786]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:06:13.3553172Z);\nlet _endTime = datetime(2026-08-24T20:06:13.3553175Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-9p7d8t7v9xz6'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:06:16.7133479Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;34da49f1-b8c4-423d-9a0d-9594ca2081bc\",\r\n \"activityId\": \"2f032998-f59c-4ae2-a694-38cc61f6ea94\",\r\n \"subActivityId\": \"554417e1-fc52-46f3-b197-f82713ea9e34\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"7e75cd47-6911-43a3-a862-d633327c1574\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;34da49f1-b8c4-423d-9a0d-9594ca2081bc ARID=2f032998-f59c-4ae2-a694-38cc61f6ea94 \u003e GW.Http.CallContext/2f032998-f59c-4ae2-a694-38cc61f6ea94 \u003e GW.ExecuteQuery/7e75cd47-6911-43a3-a862-d633327c1574 \u003e DN.FE.ExecuteQuery/554417e1-fc52-46f3-b197-f82713ea9e34)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[20:06:20.207]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:06:16.8552371Z);\nlet _endTime = datetime(2026-08-24T20:06:16.8552374Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-9p7d8t7v9xz6'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:06:20.1847522Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;8f097191-494f-4ed8-bac2-68eee8e5ff98\",\r\n \"activityId\": \"f091c01f-7b20-44b8-b04f-fb0467df4e03\",\r\n \"subActivityId\": \"7026bbd3-2c23-4814-a331-0658c9a9b209\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"c2506c2f-7658-4205-bf77-f1c430893684\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;8f097191-494f-4ed8-bac2-68eee8e5ff98 ARID=f091c01f-7b20-44b8-b04f-fb0467df4e03 \u003e GW.Http.CallContext/f091c01f-7b20-44b8-b04f-fb0467df4e03 \u003e GW.ExecuteQuery/c2506c2f-7658-4205-bf77-f1c430893684 \u003e DN.FE.ExecuteQuery/7026bbd3-2c23-4814-a331-0658c9a9b209)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:06:20.258]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:06:16.8552371Z);\nlet _endTime = datetime(2026-08-24T20:06:16.8552374Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-9p7d8t7v9xz6'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:06:20.1847522Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;8f097191-494f-4ed8-bac2-68eee8e5ff98\",\r\n \"activityId\": \"f091c01f-7b20-44b8-b04f-fb0467df4e03\",\r\n \"subActivityId\": \"7026bbd3-2c23-4814-a331-0658c9a9b209\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"c2506c2f-7658-4205-bf77-f1c430893684\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;8f097191-494f-4ed8-bac2-68eee8e5ff98 ARID=f091c01f-7b20-44b8-b04f-fb0467df4e03 \u003e GW.Http.CallContext/f091c01f-7b20-44b8-b04f-fb0467df4e03 \u003e GW.ExecuteQuery/c2506c2f-7658-4205-bf77-f1c430893684 \u003e DN.FE.ExecuteQuery/7026bbd3-2c23-4814-a331-0658c9a9b209)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[20:06:27.216]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:06:23.8328269Z);\nlet _endTime = datetime(2026-08-24T20:06:23.8328273Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-9p7d8t7v9xz6'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:06:27.1947859Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;e15ec81f-eef5-49f2-8e23-4fd3c09c50c6\",\r\n \"activityId\": \"f3d06959-0b4e-4ef2-9dda-5ba6f765681d\",\r\n \"subActivityId\": \"31d47128-d285-43b5-80f5-22705e153fd2\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"782870fe-7874-4d64-93ab-867f2156413c\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;e15ec81f-eef5-49f2-8e23-4fd3c09c50c6 ARID=f3d06959-0b4e-4ef2-9dda-5ba6f765681d \u003e GW.Http.CallContext/f3d06959-0b4e-4ef2-9dda-5ba6f765681d \u003e GW.ExecuteQuery/782870fe-7874-4d64-93ab-867f2156413c \u003e DN.FE.ExecuteQuery/31d47128-d285-43b5-80f5-22705e153fd2)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[20:06:27.336]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T20:06:23.8328269Z);\nlet _endTime = datetime(2026-08-24T20:06:23.8328273Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-9p7d8t7v9xz6'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T20:06:27.1947859Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;e15ec81f-eef5-49f2-8e23-4fd3c09c50c6\",\r\n \"activityId\": \"f3d06959-0b4e-4ef2-9dda-5ba6f765681d\",\r\n \"subActivityId\": \"31d47128-d285-43b5-80f5-22705e153fd2\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"782870fe-7874-4d64-93ab-867f2156413c\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;e15ec81f-eef5-49f2-8e23-4fd3c09c50c6 ARID=f3d06959-0b4e-4ef2-9dda-5ba6f765681d \u003e GW.Http.CallContext/f3d06959-0b4e-4ef2-9dda-5ba6f765681d \u003e GW.ExecuteQuery/782870fe-7874-4d64-93ab-867f2156413c \u003e DN.FE.ExecuteQuery/31d47128-d285-43b5-80f5-22705e153fd2)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 18:53:38 UTC · failed at e2e · test SRE can pause schedules to stop backup execution for an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000f321c0>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-6q97llkgw8zq" to finish creating: GET https://rp.j4068096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2dec0484-f10a-474f-95c1-675630f94b29 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2dec0484-f10a-474f-95c1-675630f94b29", "name": "2dec0484-f10a-474f-95c1-675630f94b29", "status": "Failed", "startTime": "2026-08-24T19:41:52.630494716Z", "endTime": "2026-08-24T20:01:00.080950134Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000f321c0>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-6q97llkgw8zq" to finish creating: GET https://rp.j4068096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2dec0484-f10a-474f-95c1-675630f94b29 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2dec0484-f10a-474f-95c1-675630f94b29", "name": "2dec0484-f10a-474f-95c1-675630f94b29", "status": "Failed", "startTime": "2026-08-24T19:41:52.630494716Z", "endTime": "2026-08-24T20:01:00.080950134Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (2) alert [hcp] KubePodNotReady fired 18:53:38 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-24T19:54:32Z Ended: 2026-08-24T20:02:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j4068096-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd512keke6njcb0ucd6lun18vonei9e-l7h8m1y3e0a2g9h", pod="router-6464bbd6fd-qbsvd", severity="warning" Description: Pod ocm-arohcpci01-2sd512keke6njcb0ucd6lun18vonei9e-l7h8m1y3e0a2g9h/router-6464bbd6fd-qbsvd has been in a non-ready state for longer than 5 minutes. alert [svc] BackendControllerRetryHotLoop fired 18:53:38 UTC · failed at alert · test [aro-hcp-observability] [svc] alert BackendControllerRetryHotLoop does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-24T19:54:46Z Ended: 2026-08-24T20:17:49Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j4068096-svc", component="backend", name="observemanagedresourcegroup", severity="warning" Description: Backend controller workqueue observemanagedresourcegroup has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work.
18:47:19	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 9ec6505	e2e	failed	★#6433 (open)	1	e2e (1) query-basic: command failed: exit status 1 18:47:19 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.002s. Expected success, but got an error: <errors.errorString \| 0xc00110c3b0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[19:51:58.031]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:51:53.2303748Z);\nlet _endTime = datetime(2026-08-24T19:51:53.2303753Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-28xbv5cwfv2p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:51:58.0038349Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;96128164-ff83-4c34-accc-2d9f110e21f0\",\r\n \"activityId\": \"8ac87f54-8740-4507-acac-3a376b12846f\",\r\n \"subActivityId\": \"dbca726b-b9d8-48a5-8295-fd974a28ccf7\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"528aab99-b3bf-4f9e-b456-a62d64410d61\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;96128164-ff83-4c34-accc-2d9f110e21f0 ARID=8ac87f54-8740-4507-acac-3a376b12846f \u003e GW.Http.CallContext/8ac87f54-8740-4507-acac-3a376b12846f \u003e GW.ExecuteQuery/528aab99-b3bf-4f9e-b456-a62d64410d61 \u003e DN.FE.ExecuteQuery/dbca726b-b9d8-48a5-8295-fd974a28ccf7)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[19:51:58.178]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:51:53.2303748Z);\nlet _endTime = datetime(2026-08-24T19:51:53.2303753Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-28xbv5cwfv2p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:51:58.0038349Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;96128164-ff83-4c34-accc-2d9f110e21f0\",\r\n \"activityId\": \"8ac87f54-8740-4507-acac-3a376b12846f\",\r\n \"subActivityId\": \"dbca726b-b9d8-48a5-8295-fd974a28ccf7\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"528aab99-b3bf-4f9e-b456-a62d64410d61\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;96128164-ff83-4c34-accc-2d9f110e21f0 ARID=8ac87f54-8740-4507-acac-3a376b12846f \u003e GW.Http.CallContext/8ac87f54-8740-4507-acac-3a376b12846f \u003e GW.ExecuteQuery/528aab99-b3bf-4f9e-b456-a62d64410d61 \u003e DN.FE.ExecuteQuery/dbca726b-b9d8-48a5-8295-fd974a28ccf7)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[19:52:03.588]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:51:58.4811848Z);\nlet _endTime = datetime(2026-08-24T19:51:58.4811858Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-28xbv5cwfv2p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:52:03.5678217Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;af1b37d1-40cf-4cf5-8093-fced46cb4c96\",\r\n \"activityId\": \"d51c8bb8-5365-4422-9f3d-bbc9cca218b5\",\r\n \"subActivityId\": \"6975476d-10b4-451b-b424-cf3649dfd75d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"8d2189a9-3ed5-4f43-814e-a113dc56eb7e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;af1b37d1-40cf-4cf5-8093-fced46cb4c96 ARID=d51c8bb8-5365-4422-9f3d-bbc9cca218b5 \u003e GW.Http.CallContext/d51c8bb8-5365-4422-9f3d-bbc9cca218b5 \u003e GW.ExecuteQuery/8d2189a9-3ed5-4f43-814e-a113dc56eb7e \u003e DN.FE.ExecuteQuery/6975476d-10b4-451b-b424-cf3649dfd75d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[19:52:03.647]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:51:58.4811848Z);\nlet _endTime = datetime(2026-08-24T19:51:58.4811858Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-28xbv5cwfv2p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:52:03.5678217Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;af1b37d1-40cf-4cf5-8093-fced46cb4c96\",\r\n \"activityId\": \"d51c8bb8-5365-4422-9f3d-bbc9cca218b5\",\r\n \"subActivityId\": \"6975476d-10b4-451b-b424-cf3649dfd75d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"8d2189a9-3ed5-4f43-814e-a113dc56eb7e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;af1b37d1-40cf-4cf5-8093-fced46cb4c96 ARID=d51c8bb8-5365-4422-9f3d-bbc9cca218b5 \u003e GW.Http.CallContext/d51c8bb8-5365-4422-9f3d-bbc9cca218b5 \u003e GW.ExecuteQuery/8d2189a9-3ed5-4f43-814e-a113dc56eb7e \u003e DN.FE.ExecuteQuery/6975476d-10b4-451b-b424-cf3649dfd75d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[19:52:14.740]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:52:09.9612940Z);\nlet _endTime = datetime(2026-08-24T19:52:09.9612946Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-28xbv5cwfv2p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:52:14.7147920Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;99f53a04-bad6-4929-931b-4d38fe5e4afd\",\r\n \"activityId\": \"b7b4b40b-e6b7-47d5-bc55-11fbc917a523\",\r\n \"subActivityId\": \"b6e1dd12-84b1-4a1c-8ebf-ce91a8b34dd6\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"8d568e6d-b055-4816-bdc6-8065304d5792\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;99f53a04-bad6-4929-931b-4d38fe5e4afd ARID=b7b4b40b-e6b7-47d5-bc55-11fbc917a523 \u003e GW.Http.CallContext/b7b4b40b-e6b7-47d5-bc55-11fbc917a523 \u003e GW.ExecuteQuery/8d568e6d-b055-4816-bdc6-8065304d5792 \u003e DN.FE.ExecuteQuery/b6e1dd12-84b1-4a1c-8ebf-ce91a8b34dd6)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[19:52:14.860]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:52:09.9612940Z);\nlet _endTime = datetime(2026-08-24T19:52:09.9612946Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-28xbv5cwfv2p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:52:14.7147920Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;99f53a04-bad6-4929-931b-4d38fe5e4afd\",\r\n \"activityId\": \"b7b4b40b-e6b7-47d5-bc55-11fbc917a523\",\r\n \"subActivityId\": \"b6e1dd12-84b1-4a1c-8ebf-ce91a8b34dd6\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"8d568e6d-b055-4816-bdc6-8065304d5792\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;99f53a04-bad6-4929-931b-4d38fe5e4afd ARID=b7b4b40b-e6b7-47d5-bc55-11fbc917a523 \u003e GW.Http.CallContext/b7b4b40b-e6b7-47d5-bc55-11fbc917a523 \u003e GW.ExecuteQuery/8d568e6d-b055-4816-bdc6-8065304d5792 \u003e DN.FE.ExecuteQuery/b6e1dd12-84b1-4a1c-8ebf-ce91a8b34dd6)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.002s. Expected success, but got an error: <errors.errorString \| 0xc00110c3b0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[19:51:58.031]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:51:53.2303748Z);\nlet _endTime = datetime(2026-08-24T19:51:53.2303753Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-28xbv5cwfv2p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:51:58.0038349Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;96128164-ff83-4c34-accc-2d9f110e21f0\",\r\n \"activityId\": \"8ac87f54-8740-4507-acac-3a376b12846f\",\r\n \"subActivityId\": \"dbca726b-b9d8-48a5-8295-fd974a28ccf7\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"528aab99-b3bf-4f9e-b456-a62d64410d61\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;96128164-ff83-4c34-accc-2d9f110e21f0 ARID=8ac87f54-8740-4507-acac-3a376b12846f \u003e GW.Http.CallContext/8ac87f54-8740-4507-acac-3a376b12846f \u003e GW.ExecuteQuery/528aab99-b3bf-4f9e-b456-a62d64410d61 \u003e DN.FE.ExecuteQuery/dbca726b-b9d8-48a5-8295-fd974a28ccf7)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[19:51:58.178]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:51:53.2303748Z);\nlet _endTime = datetime(2026-08-24T19:51:53.2303753Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-28xbv5cwfv2p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:51:58.0038349Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;96128164-ff83-4c34-accc-2d9f110e21f0\",\r\n \"activityId\": \"8ac87f54-8740-4507-acac-3a376b12846f\",\r\n \"subActivityId\": \"dbca726b-b9d8-48a5-8295-fd974a28ccf7\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"528aab99-b3bf-4f9e-b456-a62d64410d61\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;96128164-ff83-4c34-accc-2d9f110e21f0 ARID=8ac87f54-8740-4507-acac-3a376b12846f \u003e GW.Http.CallContext/8ac87f54-8740-4507-acac-3a376b12846f \u003e GW.ExecuteQuery/528aab99-b3bf-4f9e-b456-a62d64410d61 \u003e DN.FE.ExecuteQuery/dbca726b-b9d8-48a5-8295-fd974a28ccf7)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[19:52:03.588]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:51:58.4811848Z);\nlet _endTime = datetime(2026-08-24T19:51:58.4811858Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-28xbv5cwfv2p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:52:03.5678217Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;af1b37d1-40cf-4cf5-8093-fced46cb4c96\",\r\n \"activityId\": \"d51c8bb8-5365-4422-9f3d-bbc9cca218b5\",\r\n \"subActivityId\": \"6975476d-10b4-451b-b424-cf3649dfd75d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"8d2189a9-3ed5-4f43-814e-a113dc56eb7e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;af1b37d1-40cf-4cf5-8093-fced46cb4c96 ARID=d51c8bb8-5365-4422-9f3d-bbc9cca218b5 \u003e GW.Http.CallContext/d51c8bb8-5365-4422-9f3d-bbc9cca218b5 \u003e GW.ExecuteQuery/8d2189a9-3ed5-4f43-814e-a113dc56eb7e \u003e DN.FE.ExecuteQuery/6975476d-10b4-451b-b424-cf3649dfd75d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[19:52:03.647]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:51:58.4811848Z);\nlet _endTime = datetime(2026-08-24T19:51:58.4811858Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-28xbv5cwfv2p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:52:03.5678217Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;af1b37d1-40cf-4cf5-8093-fced46cb4c96\",\r\n \"activityId\": \"d51c8bb8-5365-4422-9f3d-bbc9cca218b5\",\r\n \"subActivityId\": \"6975476d-10b4-451b-b424-cf3649dfd75d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"8d2189a9-3ed5-4f43-814e-a113dc56eb7e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;af1b37d1-40cf-4cf5-8093-fced46cb4c96 ARID=d51c8bb8-5365-4422-9f3d-bbc9cca218b5 \u003e GW.Http.CallContext/d51c8bb8-5365-4422-9f3d-bbc9cca218b5 \u003e GW.ExecuteQuery/8d2189a9-3ed5-4f43-814e-a113dc56eb7e \u003e DN.FE.ExecuteQuery/6975476d-10b4-451b-b424-cf3649dfd75d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[19:52:14.740]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:52:09.9612940Z);\nlet _endTime = datetime(2026-08-24T19:52:09.9612946Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-28xbv5cwfv2p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:52:14.7147920Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;99f53a04-bad6-4929-931b-4d38fe5e4afd\",\r\n \"activityId\": \"b7b4b40b-e6b7-47d5-bc55-11fbc917a523\",\r\n \"subActivityId\": \"b6e1dd12-84b1-4a1c-8ebf-ce91a8b34dd6\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"8d568e6d-b055-4816-bdc6-8065304d5792\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;99f53a04-bad6-4929-931b-4d38fe5e4afd ARID=b7b4b40b-e6b7-47d5-bc55-11fbc917a523 \u003e GW.Http.CallContext/b7b4b40b-e6b7-47d5-bc55-11fbc917a523 \u003e GW.ExecuteQuery/8d568e6d-b055-4816-bdc6-8065304d5792 \u003e DN.FE.ExecuteQuery/b6e1dd12-84b1-4a1c-8ebf-ce91a8b34dd6)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[19:52:14.860]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:52:09.9612940Z);\nlet _endTime = datetime(2026-08-24T19:52:09.9612946Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-28xbv5cwfv2p'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:52:14.7147920Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;99f53a04-bad6-4929-931b-4d38fe5e4afd\",\r\n \"activityId\": \"b7b4b40b-e6b7-47d5-bc55-11fbc917a523\",\r\n \"subActivityId\": \"b6e1dd12-84b1-4a1c-8ebf-ce91a8b34dd6\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"8d568e6d-b055-4816-bdc6-8065304d5792\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;99f53a04-bad6-4929-931b-4d38fe5e4afd ARID=b7b4b40b-e6b7-47d5-bc55-11fbc917a523 \u003e GW.Http.CallContext/b7b4b40b-e6b7-47d5-bc55-11fbc917a523 \u003e GW.ExecuteQuery/8d568e6d-b055-4816-bdc6-8065304d5792 \u003e DN.FE.ExecuteQuery/b6e1dd12-84b1-4a1c-8ebf-ce91a8b34dd6)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ...
18:34:13	pull-ci-Azure-ARO-HCP-main-e2e-parallel head cbb6788	provision	failed	#6232 (open)	1	provision (1) ERROR CODE: DeploymentFailed; detail code AllocationFailed; detail message The VM allocation failed due to an internal error. Please retry later or try deploying to a different location.; provider Microsoft.Compute 18:34:13 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster (stamp=2) · suite step graph Show raw failure time=2026-08-24T19:10:07.135Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 description="Step cluster\n Kind: ARM\n Template: templates/mgmt-cluster.bicep\n Parameters: configurations/mgmt-cluster.tmpl.bicepparam" time=2026-08-24T19:10:08.078Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 time=2026-08-24T19:10:08.961Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 deployment=6b7c37d7d05e023c4460016a815a9ccce8683c2a1a50686369f95e5de1b45b05 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j6400512-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F6b7c37d7d05e023c4460016a815a9ccce8683c2a1a50686369f95e5de1b45b05 time=2026-08-24T19:19:10.348Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6400512-mgmt-2/providers/Microsoft.Resources/deployments/6b7c37d7d05e023c4460016a815a9ccce8683c2a1a50686369f95e5de1b45b05/operationStatuses/08584140082768889498\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6400512-mgmt-2/providers/Microsoft.Resources/deployments/cluster-3vizzf2ahtjlc\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6400512-mgmt-2/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6400512-mgmt-2/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6400512-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j6400512-mgmt-2/agentPools/userswft1\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AllocationFailed\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6400512-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft1-38184404-vmss'. Please try alternative sizes 'Standard_E16ds_v6' (zones 2, 3, 1), 'Standard_E16ds_v6' (eastus, zones 1, 2, 3), 'Standard_E16ds_v6' (eastus2, zones 2, 1, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AllocationFailed\\\",\\r\\n \\\"target\\\": \\\"3\\\",\\r\\n \\\"message\\\": \\\"The VM allocation failed due to an internal error. Please retry later or try deploying to a different location.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
18:30:02	pull-ci-Azure-ARO-HCP-main-e2e-parallel head af0ab1f	e2e	failed	★#6504 (open)	1	e2e (1) query-basic: command failed: exit status 1 18:30:02 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.003s. Expected success, but got an error: <errors.errorString \| 0xc0010363a0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[19:34:09.673]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:34:06.2000479Z);\nlet _endTime = datetime(2026-08-24T19:34:06.2000484Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-ppdpqfmxnzrc'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:34:09.6484778Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;d81cfcef-e8ba-48ea-b997-8cf06756c0b3\",\r\n \"activityId\": \"054f25c0-1fd0-49a9-99e1-26afd3a6d39a\",\r\n \"subActivityId\": \"a9e57df5-e41b-41f5-b40e-6c264fe0c14d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f615e944-9838-4c4a-a25e-26cdd6965f94\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;d81cfcef-e8ba-48ea-b997-8cf06756c0b3 ARID=054f25c0-1fd0-49a9-99e1-26afd3a6d39a \u003e GW.Http.CallContext/054f25c0-1fd0-49a9-99e1-26afd3a6d39a \u003e GW.ExecuteQuery/f615e944-9838-4c4a-a25e-26cdd6965f94 \u003e DN.FE.ExecuteQuery/a9e57df5-e41b-41f5-b40e-6c264fe0c14d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[19:34:09.728]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:34:06.2000479Z);\nlet _endTime = datetime(2026-08-24T19:34:06.2000484Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-ppdpqfmxnzrc'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:34:09.6484778Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;d81cfcef-e8ba-48ea-b997-8cf06756c0b3\",\r\n \"activityId\": \"054f25c0-1fd0-49a9-99e1-26afd3a6d39a\",\r\n \"subActivityId\": \"a9e57df5-e41b-41f5-b40e-6c264fe0c14d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f615e944-9838-4c4a-a25e-26cdd6965f94\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;d81cfcef-e8ba-48ea-b997-8cf06756c0b3 ARID=054f25c0-1fd0-49a9-99e1-26afd3a6d39a \u003e GW.Http.CallContext/054f25c0-1fd0-49a9-99e1-26afd3a6d39a \u003e GW.ExecuteQuery/f615e944-9838-4c4a-a25e-26cdd6965f94 \u003e DN.FE.ExecuteQuery/a9e57df5-e41b-41f5-b40e-6c264fe0c14d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[19:34:12.790]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:34:09.7961135Z);\nlet _endTime = datetime(2026-08-24T19:34:09.7961139Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-ppdpqfmxnzrc'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:34:12.7703289Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a00d3cd1-04de-493c-9e83-debb0c5596f9\",\r\n \"activityId\": \"f6e21dd3-10a7-425a-8be3-897128f4f105\",\r\n \"subActivityId\": \"94242df0-6fdf-4a93-831e-3661c9dff018\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"880f89f6-1e55-4a64-ad0f-c2a2d66d12dc\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a00d3cd1-04de-493c-9e83-debb0c5596f9 ARID=f6e21dd3-10a7-425a-8be3-897128f4f105 \u003e GW.Http.CallContext/f6e21dd3-10a7-425a-8be3-897128f4f105 \u003e GW.ExecuteQuery/880f89f6-1e55-4a64-ad0f-c2a2d66d12dc \u003e DN.FE.ExecuteQuery/94242df0-6fdf-4a93-831e-3661c9dff018)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[19:34:12.839]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:34:09.7961135Z);\nlet _endTime = datetime(2026-08-24T19:34:09.7961139Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-ppdpqfmxnzrc'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:34:12.7703289Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a00d3cd1-04de-493c-9e83-debb0c5596f9\",\r\n \"activityId\": \"f6e21dd3-10a7-425a-8be3-897128f4f105\",\r\n \"subActivityId\": \"94242df0-6fdf-4a93-831e-3661c9dff018\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"880f89f6-1e55-4a64-ad0f-c2a2d66d12dc\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a00d3cd1-04de-493c-9e83-debb0c5596f9 ARID=f6e21dd3-10a7-425a-8be3-897128f4f105 \u003e GW.Http.CallContext/f6e21dd3-10a7-425a-8be3-897128f4f105 \u003e GW.ExecuteQuery/880f89f6-1e55-4a64-ad0f-c2a2d66d12dc \u003e DN.FE.ExecuteQuery/94242df0-6fdf-4a93-831e-3661c9dff018)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[19:34:18.965]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:34:15.9977616Z);\nlet _endTime = datetime(2026-08-24T19:34:15.9977620Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-ppdpqfmxnzrc'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:34:18.9443276Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;15af706b-c55b-4e65-86ce-80b53c2c1b4c\",\r\n \"activityId\": \"3d775bce-17b2-46ea-9bb0-d1bc0ef2f068\",\r\n \"subActivityId\": \"6e75bb4c-3bcf-4e6f-9051-548216fd4a21\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f26336ef-c50d-4796-9b1c-509b00e04b53\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;15af706b-c55b-4e65-86ce-80b53c2c1b4c ARID=3d775bce-17b2-46ea-9bb0-d1bc0ef2f068 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/d52dede4-e619-4aff-ae18-79e397067d8c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/d52dede4-e619-4aff-ae18-79e397067d8c \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/f26336ef-c50d-4796-9b1c-509b00e04b53 \u003e DN.FE.ExecuteQuery/6e75bb4c-3bcf-4e6f-9051-548216fd4a21)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[19:34:19.063]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:34:15.9977616Z);\nlet _endTime = datetime(2026-08-24T19:34:15.9977620Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-ppdpqfmxnzrc'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:34:18.9443276Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;15af706b-c55b-4e65-86ce-80b53c2c1b4c\",\r\n \"activityId\": \"3d775bce-17b2-46ea-9bb0-d1bc0ef2f068\",\r\n \"subActivityId\": \"6e75bb4c-3bcf-4e6f-9051-548216fd4a21\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f26336ef-c50d-4796-9b1c-509b00e04b53\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;15af706b-c55b-4e65-86ce-80b53c2c1b4c ARID=3d775bce-17b2-46ea-9bb0-d1bc0ef2f068 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/d52dede4-e619-4aff-ae18-79e397067d8c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/d52dede4-e619-4aff-ae18-79e397067d8c \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/f26336ef-c50d-4796-9b1c-509b00e04b53 \u003e DN.FE.ExecuteQuery/6e75bb4c-3bcf-4e6f-9051-548216fd4a21)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.003s. Expected success, but got an error: <errors.errorString \| 0xc0010363a0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[19:34:09.673]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:34:06.2000479Z);\nlet _endTime = datetime(2026-08-24T19:34:06.2000484Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-ppdpqfmxnzrc'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:34:09.6484778Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;d81cfcef-e8ba-48ea-b997-8cf06756c0b3\",\r\n \"activityId\": \"054f25c0-1fd0-49a9-99e1-26afd3a6d39a\",\r\n \"subActivityId\": \"a9e57df5-e41b-41f5-b40e-6c264fe0c14d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f615e944-9838-4c4a-a25e-26cdd6965f94\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;d81cfcef-e8ba-48ea-b997-8cf06756c0b3 ARID=054f25c0-1fd0-49a9-99e1-26afd3a6d39a \u003e GW.Http.CallContext/054f25c0-1fd0-49a9-99e1-26afd3a6d39a \u003e GW.ExecuteQuery/f615e944-9838-4c4a-a25e-26cdd6965f94 \u003e DN.FE.ExecuteQuery/a9e57df5-e41b-41f5-b40e-6c264fe0c14d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[19:34:09.728]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:34:06.2000479Z);\nlet _endTime = datetime(2026-08-24T19:34:06.2000484Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-ppdpqfmxnzrc'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:34:09.6484778Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;d81cfcef-e8ba-48ea-b997-8cf06756c0b3\",\r\n \"activityId\": \"054f25c0-1fd0-49a9-99e1-26afd3a6d39a\",\r\n \"subActivityId\": \"a9e57df5-e41b-41f5-b40e-6c264fe0c14d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f615e944-9838-4c4a-a25e-26cdd6965f94\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;d81cfcef-e8ba-48ea-b997-8cf06756c0b3 ARID=054f25c0-1fd0-49a9-99e1-26afd3a6d39a \u003e GW.Http.CallContext/054f25c0-1fd0-49a9-99e1-26afd3a6d39a \u003e GW.ExecuteQuery/f615e944-9838-4c4a-a25e-26cdd6965f94 \u003e DN.FE.ExecuteQuery/a9e57df5-e41b-41f5-b40e-6c264fe0c14d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[19:34:12.790]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:34:09.7961135Z);\nlet _endTime = datetime(2026-08-24T19:34:09.7961139Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-ppdpqfmxnzrc'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:34:12.7703289Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a00d3cd1-04de-493c-9e83-debb0c5596f9\",\r\n \"activityId\": \"f6e21dd3-10a7-425a-8be3-897128f4f105\",\r\n \"subActivityId\": \"94242df0-6fdf-4a93-831e-3661c9dff018\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"880f89f6-1e55-4a64-ad0f-c2a2d66d12dc\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a00d3cd1-04de-493c-9e83-debb0c5596f9 ARID=f6e21dd3-10a7-425a-8be3-897128f4f105 \u003e GW.Http.CallContext/f6e21dd3-10a7-425a-8be3-897128f4f105 \u003e GW.ExecuteQuery/880f89f6-1e55-4a64-ad0f-c2a2d66d12dc \u003e DN.FE.ExecuteQuery/94242df0-6fdf-4a93-831e-3661c9dff018)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[19:34:12.839]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:34:09.7961135Z);\nlet _endTime = datetime(2026-08-24T19:34:09.7961139Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-ppdpqfmxnzrc'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:34:12.7703289Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a00d3cd1-04de-493c-9e83-debb0c5596f9\",\r\n \"activityId\": \"f6e21dd3-10a7-425a-8be3-897128f4f105\",\r\n \"subActivityId\": \"94242df0-6fdf-4a93-831e-3661c9dff018\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"880f89f6-1e55-4a64-ad0f-c2a2d66d12dc\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a00d3cd1-04de-493c-9e83-debb0c5596f9 ARID=f6e21dd3-10a7-425a-8be3-897128f4f105 \u003e GW.Http.CallContext/f6e21dd3-10a7-425a-8be3-897128f4f105 \u003e GW.ExecuteQuery/880f89f6-1e55-4a64-ad0f-c2a2d66d12dc \u003e DN.FE.ExecuteQuery/94242df0-6fdf-4a93-831e-3661c9dff018)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[19:34:18.965]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:34:15.9977616Z);\nlet _endTime = datetime(2026-08-24T19:34:15.9977620Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-ppdpqfmxnzrc'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:34:18.9443276Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;15af706b-c55b-4e65-86ce-80b53c2c1b4c\",\r\n \"activityId\": \"3d775bce-17b2-46ea-9bb0-d1bc0ef2f068\",\r\n \"subActivityId\": \"6e75bb4c-3bcf-4e6f-9051-548216fd4a21\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f26336ef-c50d-4796-9b1c-509b00e04b53\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;15af706b-c55b-4e65-86ce-80b53c2c1b4c ARID=3d775bce-17b2-46ea-9bb0-d1bc0ef2f068 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/d52dede4-e619-4aff-ae18-79e397067d8c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/d52dede4-e619-4aff-ae18-79e397067d8c \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/f26336ef-c50d-4796-9b1c-509b00e04b53 \u003e DN.FE.ExecuteQuery/6e75bb4c-3bcf-4e6f-9051-548216fd4a21)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[19:34:19.063]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:34:15.9977616Z);\nlet _endTime = datetime(2026-08-24T19:34:15.9977620Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-ppdpqfmxnzrc'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:34:18.9443276Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;15af706b-c55b-4e65-86ce-80b53c2c1b4c\",\r\n \"activityId\": \"3d775bce-17b2-46ea-9bb0-d1bc0ef2f068\",\r\n \"subActivityId\": \"6e75bb4c-3bcf-4e6f-9051-548216fd4a21\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f26336ef-c50d-4796-9b1c-509b00e04b53\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;15af706b-c55b-4e65-86ce-80b53c2c1b4c ARID=3d775bce-17b2-46ea-9bb0-d1bc0ef2f068 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/d52dede4-e619-4aff-ae18-79e397067d8c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/d52dede4-e619-4aff-ae18-79e397067d8c \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/f26336ef-c50d-4796-9b1c-509b00e04b53 \u003e DN.FE.ExecuteQuery/6e75bb4c-3bcf-4e6f-9051-548216fd4a21)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ...
18:15:46	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head af0ab1f	provision	failed	#6504 (open)	1	provision (1) ERROR CODE: DeploymentNotFound 18:15:46 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Velero/management/output (stamp=1) · suite step graph Show raw failure time=2026-08-24T18:57:37.594Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=management step=output stamp=1 description="Step output\n Kind: ARM\n Template: ./../dev-infrastructure/templates/output-mgmt.bicep\n Parameters: ./../dev-infrastructure/configurations/output-mgmt.tmpl.bicepparam" time=2026-08-24T18:57:37.979Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=management step=output stamp=1 time=2026-08-24T18:57:38.350Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=management step=output stamp=1 deployment=cfd650f4557893b92396842785417ac4f900952cc40f35ef046b64198c66a837 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j8797056-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fcfd650f4557893b92396842785417ac4f900952cc40f35ef046b64198c66a837 time=2026-08-24T18:57:38.400Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=management step=output stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8797056-mgmt-1/providers/Microsoft.Resources/deployments/cfd650f4557893b92396842785417ac4f900952cc40f35ef046b64198c66a837/operationStatuses/08584140090272578417\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DeploymentNotFound\",\n \"message\": \"Deployment 'cfd650f4557893b92396842785417ac4f900952cc40f35ef046b64198c66a837' could not be found.\"\n }\n}\n--------------------------------------------------------------------------------\n"
18:09:09	pull-ci-Azure-ARO-HCP-main-e2e-parallel head dd56f9b	e2e	failed	★#6661 (open)	1	e2e (1) query-basic: command failed: exit status 1 18:09:09 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 619.631s. Expected success, but got an error: <errors.errorString \| 0xc0030681a0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[19:23:44.062]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:23:40.1519515Z);\nlet _endTime = datetime(2026-08-24T19:23:40.1519520Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-rnpv84j6wl7r'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:23:44.0398678Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;3d7a5f03-f3c0-4765-8dfb-97bafe8a515f\",\r\n \"activityId\": \"8deb662e-4ed5-4295-b951-12b5409e7ea3\",\r\n \"subActivityId\": \"0cf0153c-3828-4601-9bcc-7a115de17b07\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"5881bf3c-b1e1-47fb-b608-3ae541bc20d5\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;3d7a5f03-f3c0-4765-8dfb-97bafe8a515f ARID=8deb662e-4ed5-4295-b951-12b5409e7ea3 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/5e0df9ce-766c-4b6f-8970-5b75e7392f9a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/5e0df9ce-766c-4b6f-8970-5b75e7392f9a \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/5881bf3c-b1e1-47fb-b608-3ae541bc20d5 \u003e DN.FE.ExecuteQuery/0cf0153c-3828-4601-9bcc-7a115de17b07)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[19:23:44.117]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:23:40.1519515Z);\nlet _endTime = datetime(2026-08-24T19:23:40.1519520Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-rnpv84j6wl7r'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:23:44.0398678Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;3d7a5f03-f3c0-4765-8dfb-97bafe8a515f\",\r\n \"activityId\": \"8deb662e-4ed5-4295-b951-12b5409e7ea3\",\r\n \"subActivityId\": \"0cf0153c-3828-4601-9bcc-7a115de17b07\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"5881bf3c-b1e1-47fb-b608-3ae541bc20d5\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;3d7a5f03-f3c0-4765-8dfb-97bafe8a515f ARID=8deb662e-4ed5-4295-b951-12b5409e7ea3 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/5e0df9ce-766c-4b6f-8970-5b75e7392f9a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/5e0df9ce-766c-4b6f-8970-5b75e7392f9a \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/5881bf3c-b1e1-47fb-b608-3ae541bc20d5 \u003e DN.FE.ExecuteQuery/0cf0153c-3828-4601-9bcc-7a115de17b07)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[19:23:47.256]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:23:44.1887769Z);\nlet _endTime = datetime(2026-08-24T19:23:44.1887774Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-rnpv84j6wl7r'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:23:47.2358741Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;8baad5fe-2a10-4dfd-9257-59e57e128d07\",\r\n \"activityId\": \"1827bac2-5232-4e70-8c09-35636e80a789\",\r\n \"subActivityId\": \"fa1cc2d9-248b-49c8-bc06-1f5f35c373f0\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"ecdbfa8e-92e7-472a-91fe-d06a9fbd8588\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;8baad5fe-2a10-4dfd-9257-59e57e128d07 ARID=1827bac2-5232-4e70-8c09-35636e80a789 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/992406a7-67e7-4944-9ccf-24ecdf840578 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/992406a7-67e7-4944-9ccf-24ecdf840578 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/ecdbfa8e-92e7-472a-91fe-d06a9fbd8588 \u003e DN.FE.ExecuteQuery/fa1cc2d9-248b-49c8-bc06-1f5f35c373f0)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[19:23:47.310]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:23:44.1887769Z);\nlet _endTime = datetime(2026-08-24T19:23:44.1887774Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-rnpv84j6wl7r'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:23:47.2358741Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;8baad5fe-2a10-4dfd-9257-59e57e128d07\",\r\n \"activityId\": \"1827bac2-5232-4e70-8c09-35636e80a789\",\r\n \"subActivityId\": \"fa1cc2d9-248b-49c8-bc06-1f5f35c373f0\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"ecdbfa8e-92e7-472a-91fe-d06a9fbd8588\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;8baad5fe-2a10-4dfd-9257-59e57e128d07 ARID=1827bac2-5232-4e70-8c09-35636e80a789 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/992406a7-67e7-4944-9ccf-24ecdf840578 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/992406a7-67e7-4944-9ccf-24ecdf840578 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/ecdbfa8e-92e7-472a-91fe-d06a9fbd8588 \u003e DN.FE.ExecuteQuery/fa1cc2d9-248b-49c8-bc06-1f5f35c373f0)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[19:23:53.957]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:23:50.5451272Z);\nlet _endTime = datetime(2026-08-24T19:23:50.5451277Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-rnpv84j6wl7r'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:23:53.9300116Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;90f6c849-5516-4655-952a-2d8ba0fadb0e\",\r\n \"activityId\": \"69d47628-ee22-4c1a-8e44-ac08996ff22b\",\r\n \"subActivityId\": \"bcf84e1b-d7ce-46c7-a057-04c241df794c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"4f989b02-f73a-48b8-9cc5-b3a9e0cd2026\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;90f6c849-5516-4655-952a-2d8ba0fadb0e ARID=69d47628-ee22-4c1a-8e44-ac08996ff22b \u003e GW.Http.CallContext/69d47628-ee22-4c1a-8e44-ac08996ff22b \u003e GW.ExecuteQuery/4f989b02-f73a-48b8-9cc5-b3a9e0cd2026 \u003e DN.FE.ExecuteQuery/bcf84e1b-d7ce-46c7-a057-04c241df794c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[19:23:54.071]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:23:50.5451272Z);\nlet _endTime = datetime(2026-08-24T19:23:50.5451277Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-rnpv84j6wl7r'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:23:53.9300116Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;90f6c849-5516-4655-952a-2d8ba0fadb0e\",\r\n \"activityId\": \"69d47628-ee22-4c1a-8e44-ac08996ff22b\",\r\n \"subActivityId\": \"bcf84e1b-d7ce-46c7-a057-04c241df794c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"4f989b02-f73a-48b8-9cc5-b3a9e0cd2026\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;90f6c849-5516-4655-952a-2d8ba0fadb0e ARID=69d47628-ee22-4c1a-8e44-ac08996ff22b \u003e GW.Http.CallContext/69d47628-ee22-4c1a-8e44-ac08996ff22b \u003e GW.ExecuteQuery/4f989b02-f73a-48b8-9cc5-b3a9e0cd2026 \u003e DN.FE.ExecuteQuery/bcf84e1b-d7ce-46c7-a057-04c241df794c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 619.631s. Expected success, but got an error: <errors.errorString \| 0xc0030681a0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[19:23:44.062]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:23:40.1519515Z);\nlet _endTime = datetime(2026-08-24T19:23:40.1519520Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-rnpv84j6wl7r'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:23:44.0398678Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;3d7a5f03-f3c0-4765-8dfb-97bafe8a515f\",\r\n \"activityId\": \"8deb662e-4ed5-4295-b951-12b5409e7ea3\",\r\n \"subActivityId\": \"0cf0153c-3828-4601-9bcc-7a115de17b07\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"5881bf3c-b1e1-47fb-b608-3ae541bc20d5\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;3d7a5f03-f3c0-4765-8dfb-97bafe8a515f ARID=8deb662e-4ed5-4295-b951-12b5409e7ea3 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/5e0df9ce-766c-4b6f-8970-5b75e7392f9a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/5e0df9ce-766c-4b6f-8970-5b75e7392f9a \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/5881bf3c-b1e1-47fb-b608-3ae541bc20d5 \u003e DN.FE.ExecuteQuery/0cf0153c-3828-4601-9bcc-7a115de17b07)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[19:23:44.117]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:23:40.1519515Z);\nlet _endTime = datetime(2026-08-24T19:23:40.1519520Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-rnpv84j6wl7r'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:23:44.0398678Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;3d7a5f03-f3c0-4765-8dfb-97bafe8a515f\",\r\n \"activityId\": \"8deb662e-4ed5-4295-b951-12b5409e7ea3\",\r\n \"subActivityId\": \"0cf0153c-3828-4601-9bcc-7a115de17b07\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"5881bf3c-b1e1-47fb-b608-3ae541bc20d5\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;3d7a5f03-f3c0-4765-8dfb-97bafe8a515f ARID=8deb662e-4ed5-4295-b951-12b5409e7ea3 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/5e0df9ce-766c-4b6f-8970-5b75e7392f9a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/5e0df9ce-766c-4b6f-8970-5b75e7392f9a \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/5881bf3c-b1e1-47fb-b608-3ae541bc20d5 \u003e DN.FE.ExecuteQuery/0cf0153c-3828-4601-9bcc-7a115de17b07)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[19:23:47.256]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:23:44.1887769Z);\nlet _endTime = datetime(2026-08-24T19:23:44.1887774Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-rnpv84j6wl7r'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:23:47.2358741Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;8baad5fe-2a10-4dfd-9257-59e57e128d07\",\r\n \"activityId\": \"1827bac2-5232-4e70-8c09-35636e80a789\",\r\n \"subActivityId\": \"fa1cc2d9-248b-49c8-bc06-1f5f35c373f0\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"ecdbfa8e-92e7-472a-91fe-d06a9fbd8588\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;8baad5fe-2a10-4dfd-9257-59e57e128d07 ARID=1827bac2-5232-4e70-8c09-35636e80a789 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/992406a7-67e7-4944-9ccf-24ecdf840578 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/992406a7-67e7-4944-9ccf-24ecdf840578 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/ecdbfa8e-92e7-472a-91fe-d06a9fbd8588 \u003e DN.FE.ExecuteQuery/fa1cc2d9-248b-49c8-bc06-1f5f35c373f0)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[19:23:47.310]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:23:44.1887769Z);\nlet _endTime = datetime(2026-08-24T19:23:44.1887774Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-rnpv84j6wl7r'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:23:47.2358741Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;8baad5fe-2a10-4dfd-9257-59e57e128d07\",\r\n \"activityId\": \"1827bac2-5232-4e70-8c09-35636e80a789\",\r\n \"subActivityId\": \"fa1cc2d9-248b-49c8-bc06-1f5f35c373f0\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"ecdbfa8e-92e7-472a-91fe-d06a9fbd8588\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;8baad5fe-2a10-4dfd-9257-59e57e128d07 ARID=1827bac2-5232-4e70-8c09-35636e80a789 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/992406a7-67e7-4944-9ccf-24ecdf840578 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/992406a7-67e7-4944-9ccf-24ecdf840578 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/ecdbfa8e-92e7-472a-91fe-d06a9fbd8588 \u003e DN.FE.ExecuteQuery/fa1cc2d9-248b-49c8-bc06-1f5f35c373f0)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[19:23:53.957]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:23:50.5451272Z);\nlet _endTime = datetime(2026-08-24T19:23:50.5451277Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-rnpv84j6wl7r'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:23:53.9300116Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;90f6c849-5516-4655-952a-2d8ba0fadb0e\",\r\n \"activityId\": \"69d47628-ee22-4c1a-8e44-ac08996ff22b\",\r\n \"subActivityId\": \"bcf84e1b-d7ce-46c7-a057-04c241df794c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"4f989b02-f73a-48b8-9cc5-b3a9e0cd2026\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;90f6c849-5516-4655-952a-2d8ba0fadb0e ARID=69d47628-ee22-4c1a-8e44-ac08996ff22b \u003e GW.Http.CallContext/69d47628-ee22-4c1a-8e44-ac08996ff22b \u003e GW.ExecuteQuery/4f989b02-f73a-48b8-9cc5-b3a9e0cd2026 \u003e DN.FE.ExecuteQuery/bcf84e1b-d7ce-46c7-a057-04c241df794c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[19:23:54.071]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:23:50.5451272Z);\nlet _endTime = datetime(2026-08-24T19:23:50.5451277Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-rnpv84j6wl7r'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:23:53.9300116Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;90f6c849-5516-4655-952a-2d8ba0fadb0e\",\r\n \"activityId\": \"69d47628-ee22-4c1a-8e44-ac08996ff22b\",\r\n \"subActivityId\": \"bcf84e1b-d7ce-46c7-a057-04c241df794c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"4f989b02-f73a-48b8-9cc5-b3a9e0cd2026\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;90f6c849-5516-4655-952a-2d8ba0fadb0e ARID=69d47628-ee22-4c1a-8e44-ac08996ff22b \u003e GW.Http.CallContext/69d47628-ee22-4c1a-8e44-ac08996ff22b \u003e GW.ExecuteQuery/4f989b02-f73a-48b8-9cc5-b3a9e0cd2026 \u003e DN.FE.ExecuteQuery/bcf84e1b-d7ce-46c7-a057-04c241df794c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ...
17:58:06	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 09ab1aa	e2e	failed	★#6529 (open)	1	e2e (1) query-basic: command failed: exit status 1 17:58:06 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.006s. Expected success, but got an error: <errors.errorString \| 0xc00113a410>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[19:10:20.778]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:10:17.4484437Z);\nlet _endTime = datetime(2026-08-24T19:10:17.4484442Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-jz5mh44r448g'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:10:20.7536026Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;5dc9f478-db70-4e96-8724-131234c12955\",\r\n \"activityId\": \"7149451f-d11b-433b-9a3b-571a10a45165\",\r\n \"subActivityId\": \"fb3e90ce-fa8e-4e9c-bfa9-a3976da8056b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"6d73438d-0db4-4318-8b8c-12ab62681dba\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;5dc9f478-db70-4e96-8724-131234c12955 ARID=7149451f-d11b-433b-9a3b-571a10a45165 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/3efe508e-a599-46c0-ba0a-5405fc13be3c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/3efe508e-a599-46c0-ba0a-5405fc13be3c \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/6d73438d-0db4-4318-8b8c-12ab62681dba \u003e DN.FE.ExecuteQuery/fb3e90ce-fa8e-4e9c-bfa9-a3976da8056b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[19:10:20.833]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:10:17.4484437Z);\nlet _endTime = datetime(2026-08-24T19:10:17.4484442Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-jz5mh44r448g'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:10:20.7536026Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;5dc9f478-db70-4e96-8724-131234c12955\",\r\n \"activityId\": \"7149451f-d11b-433b-9a3b-571a10a45165\",\r\n \"subActivityId\": \"fb3e90ce-fa8e-4e9c-bfa9-a3976da8056b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"6d73438d-0db4-4318-8b8c-12ab62681dba\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;5dc9f478-db70-4e96-8724-131234c12955 ARID=7149451f-d11b-433b-9a3b-571a10a45165 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/3efe508e-a599-46c0-ba0a-5405fc13be3c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/3efe508e-a599-46c0-ba0a-5405fc13be3c \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/6d73438d-0db4-4318-8b8c-12ab62681dba \u003e DN.FE.ExecuteQuery/fb3e90ce-fa8e-4e9c-bfa9-a3976da8056b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[19:10:24.316]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:10:20.8951251Z);\nlet _endTime = datetime(2026-08-24T19:10:20.8951256Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-jz5mh44r448g'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:10:24.2915194Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;178dc16e-33e1-4eaf-8836-e5b38c257d27\",\r\n \"activityId\": \"745e2dd1-7af3-4ad2-9908-aea0f029825a\",\r\n \"subActivityId\": \"6e56d219-d4a4-4bdb-9237-cec7d05218d2\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"433a1557-6ae3-4619-9501-8e588d6e9ab9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;178dc16e-33e1-4eaf-8836-e5b38c257d27 ARID=745e2dd1-7af3-4ad2-9908-aea0f029825a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/c030aa9b-1740-40eb-adb1-25e2aeb0a8d9 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/c030aa9b-1740-40eb-adb1-25e2aeb0a8d9 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/433a1557-6ae3-4619-9501-8e588d6e9ab9 \u003e DN.FE.ExecuteQuery/6e56d219-d4a4-4bdb-9237-cec7d05218d2)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[19:10:24.372]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:10:20.8951251Z);\nlet _endTime = datetime(2026-08-24T19:10:20.8951256Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-jz5mh44r448g'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:10:24.2915194Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;178dc16e-33e1-4eaf-8836-e5b38c257d27\",\r\n \"activityId\": \"745e2dd1-7af3-4ad2-9908-aea0f029825a\",\r\n \"subActivityId\": \"6e56d219-d4a4-4bdb-9237-cec7d05218d2\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"433a1557-6ae3-4619-9501-8e588d6e9ab9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;178dc16e-33e1-4eaf-8836-e5b38c257d27 ARID=745e2dd1-7af3-4ad2-9908-aea0f029825a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/c030aa9b-1740-40eb-adb1-25e2aeb0a8d9 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/c030aa9b-1740-40eb-adb1-25e2aeb0a8d9 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/433a1557-6ae3-4619-9501-8e588d6e9ab9 \u003e DN.FE.ExecuteQuery/6e56d219-d4a4-4bdb-9237-cec7d05218d2)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[19:10:30.712]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:10:27.5126036Z);\nlet _endTime = datetime(2026-08-24T19:10:27.5126041Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-jz5mh44r448g'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:10:30.6904943Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;2d47fae2-119e-4455-b4d0-4950975b7d93\",\r\n \"activityId\": \"285a1bcb-0b9a-4c59-a17c-c879f622d59f\",\r\n \"subActivityId\": \"2eebac36-96de-45fd-8914-3686b027b36c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"011666bc-adc9-46a4-a613-43528a4b1482\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;2d47fae2-119e-4455-b4d0-4950975b7d93 ARID=285a1bcb-0b9a-4c59-a17c-c879f622d59f \u003e GW.Http.CallContext/285a1bcb-0b9a-4c59-a17c-c879f622d59f \u003e GW.ExecuteQuery/011666bc-adc9-46a4-a613-43528a4b1482 \u003e DN.FE.ExecuteQuery/2eebac36-96de-45fd-8914-3686b027b36c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[19:10:30.817]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:10:27.5126036Z);\nlet _endTime = datetime(2026-08-24T19:10:27.5126041Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-jz5mh44r448g'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:10:30.6904943Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;2d47fae2-119e-4455-b4d0-4950975b7d93\",\r\n \"activityId\": \"285a1bcb-0b9a-4c59-a17c-c879f622d59f\",\r\n \"subActivityId\": \"2eebac36-96de-45fd-8914-3686b027b36c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"011666bc-adc9-46a4-a613-43528a4b1482\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;2d47fae2-119e-4455-b4d0-4950975b7d93 ARID=285a1bcb-0b9a-4c59-a17c-c879f622d59f \u003e GW.Http.CallContext/285a1bcb-0b9a-4c59-a17c-c879f622d59f \u003e GW.ExecuteQuery/011666bc-adc9-46a4-a613-43528a4b1482 \u003e DN.FE.ExecuteQuery/2eebac36-96de-45fd-8914-3686b027b36c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.006s. Expected success, but got an error: <errors.errorString \| 0xc00113a410>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[19:10:20.778]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:10:17.4484437Z);\nlet _endTime = datetime(2026-08-24T19:10:17.4484442Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-jz5mh44r448g'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:10:20.7536026Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;5dc9f478-db70-4e96-8724-131234c12955\",\r\n \"activityId\": \"7149451f-d11b-433b-9a3b-571a10a45165\",\r\n \"subActivityId\": \"fb3e90ce-fa8e-4e9c-bfa9-a3976da8056b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"6d73438d-0db4-4318-8b8c-12ab62681dba\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;5dc9f478-db70-4e96-8724-131234c12955 ARID=7149451f-d11b-433b-9a3b-571a10a45165 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/3efe508e-a599-46c0-ba0a-5405fc13be3c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/3efe508e-a599-46c0-ba0a-5405fc13be3c \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/6d73438d-0db4-4318-8b8c-12ab62681dba \u003e DN.FE.ExecuteQuery/fb3e90ce-fa8e-4e9c-bfa9-a3976da8056b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[19:10:20.833]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:10:17.4484437Z);\nlet _endTime = datetime(2026-08-24T19:10:17.4484442Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-jz5mh44r448g'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:10:20.7536026Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;5dc9f478-db70-4e96-8724-131234c12955\",\r\n \"activityId\": \"7149451f-d11b-433b-9a3b-571a10a45165\",\r\n \"subActivityId\": \"fb3e90ce-fa8e-4e9c-bfa9-a3976da8056b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"6d73438d-0db4-4318-8b8c-12ab62681dba\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;5dc9f478-db70-4e96-8724-131234c12955 ARID=7149451f-d11b-433b-9a3b-571a10a45165 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/3efe508e-a599-46c0-ba0a-5405fc13be3c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/3efe508e-a599-46c0-ba0a-5405fc13be3c \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/6d73438d-0db4-4318-8b8c-12ab62681dba \u003e DN.FE.ExecuteQuery/fb3e90ce-fa8e-4e9c-bfa9-a3976da8056b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[19:10:24.316]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:10:20.8951251Z);\nlet _endTime = datetime(2026-08-24T19:10:20.8951256Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-jz5mh44r448g'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:10:24.2915194Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;178dc16e-33e1-4eaf-8836-e5b38c257d27\",\r\n \"activityId\": \"745e2dd1-7af3-4ad2-9908-aea0f029825a\",\r\n \"subActivityId\": \"6e56d219-d4a4-4bdb-9237-cec7d05218d2\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"433a1557-6ae3-4619-9501-8e588d6e9ab9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;178dc16e-33e1-4eaf-8836-e5b38c257d27 ARID=745e2dd1-7af3-4ad2-9908-aea0f029825a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/c030aa9b-1740-40eb-adb1-25e2aeb0a8d9 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/c030aa9b-1740-40eb-adb1-25e2aeb0a8d9 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/433a1557-6ae3-4619-9501-8e588d6e9ab9 \u003e DN.FE.ExecuteQuery/6e56d219-d4a4-4bdb-9237-cec7d05218d2)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[19:10:24.372]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:10:20.8951251Z);\nlet _endTime = datetime(2026-08-24T19:10:20.8951256Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-jz5mh44r448g'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:10:24.2915194Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;178dc16e-33e1-4eaf-8836-e5b38c257d27\",\r\n \"activityId\": \"745e2dd1-7af3-4ad2-9908-aea0f029825a\",\r\n \"subActivityId\": \"6e56d219-d4a4-4bdb-9237-cec7d05218d2\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"433a1557-6ae3-4619-9501-8e588d6e9ab9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;178dc16e-33e1-4eaf-8836-e5b38c257d27 ARID=745e2dd1-7af3-4ad2-9908-aea0f029825a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/c030aa9b-1740-40eb-adb1-25e2aeb0a8d9 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/c030aa9b-1740-40eb-adb1-25e2aeb0a8d9 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/433a1557-6ae3-4619-9501-8e588d6e9ab9 \u003e DN.FE.ExecuteQuery/6e56d219-d4a4-4bdb-9237-cec7d05218d2)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[19:10:30.712]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:10:27.5126036Z);\nlet _endTime = datetime(2026-08-24T19:10:27.5126041Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-jz5mh44r448g'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:10:30.6904943Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;2d47fae2-119e-4455-b4d0-4950975b7d93\",\r\n \"activityId\": \"285a1bcb-0b9a-4c59-a17c-c879f622d59f\",\r\n \"subActivityId\": \"2eebac36-96de-45fd-8914-3686b027b36c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"011666bc-adc9-46a4-a613-43528a4b1482\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;2d47fae2-119e-4455-b4d0-4950975b7d93 ARID=285a1bcb-0b9a-4c59-a17c-c879f622d59f \u003e GW.Http.CallContext/285a1bcb-0b9a-4c59-a17c-c879f622d59f \u003e GW.ExecuteQuery/011666bc-adc9-46a4-a613-43528a4b1482 \u003e DN.FE.ExecuteQuery/2eebac36-96de-45fd-8914-3686b027b36c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[19:10:30.817]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T19:10:27.5126036Z);\nlet _endTime = datetime(2026-08-24T19:10:27.5126041Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-jz5mh44r448g'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T19:10:30.6904943Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;2d47fae2-119e-4455-b4d0-4950975b7d93\",\r\n \"activityId\": \"285a1bcb-0b9a-4c59-a17c-c879f622d59f\",\r\n \"subActivityId\": \"2eebac36-96de-45fd-8914-3686b027b36c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"011666bc-adc9-46a4-a613-43528a4b1482\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;2d47fae2-119e-4455-b4d0-4950975b7d93 ARID=285a1bcb-0b9a-4c59-a17c-c879f622d59f \u003e GW.Http.CallContext/285a1bcb-0b9a-4c59-a17c-c879f622d59f \u003e GW.ExecuteQuery/011666bc-adc9-46a4-a613-43528a4b1482 \u003e DN.FE.ExecuteQuery/2eebac36-96de-45fd-8914-3686b027b36c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ...
17:33:55	pull-ci-Azure-ARO-HCP-main-e2e-parallel head a660b6e	e2e	failed	★#6660 (open)	1	e2e (1) query-basic: command failed: exit status 1 17:33:55 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.009s. Expected success, but got an error: <errors.errorString \| 0xc001b003c0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[18:43:29.806]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:43:26.1018327Z);\nlet _endTime = datetime(2026-08-24T18:43:26.1018334Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-76lkckgjnhnb'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:43:29.7846996Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;9c4baa00-2868-44b4-9825-b777d40ca5d5\",\r\n \"activityId\": \"7aaddfd7-cd62-422f-9ff8-20bc627c5167\",\r\n \"subActivityId\": \"8b05892c-23bc-47d2-aec0-9c031e80fc1b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"214720a0-3b18-41d3-91a6-6d3d1da6e215\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;9c4baa00-2868-44b4-9825-b777d40ca5d5 ARID=7aaddfd7-cd62-422f-9ff8-20bc627c5167 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/2e60780d-ba60-40ae-9636-d9c2055ea7e4 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/2e60780d-ba60-40ae-9636-d9c2055ea7e4 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/214720a0-3b18-41d3-91a6-6d3d1da6e215 \u003e DN.FE.ExecuteQuery/8b05892c-23bc-47d2-aec0-9c031e80fc1b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:43:29.857]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:43:26.1018327Z);\nlet _endTime = datetime(2026-08-24T18:43:26.1018334Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-76lkckgjnhnb'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:43:29.7846996Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;9c4baa00-2868-44b4-9825-b777d40ca5d5\",\r\n \"activityId\": \"7aaddfd7-cd62-422f-9ff8-20bc627c5167\",\r\n \"subActivityId\": \"8b05892c-23bc-47d2-aec0-9c031e80fc1b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"214720a0-3b18-41d3-91a6-6d3d1da6e215\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;9c4baa00-2868-44b4-9825-b777d40ca5d5 ARID=7aaddfd7-cd62-422f-9ff8-20bc627c5167 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/2e60780d-ba60-40ae-9636-d9c2055ea7e4 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/2e60780d-ba60-40ae-9636-d9c2055ea7e4 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/214720a0-3b18-41d3-91a6-6d3d1da6e215 \u003e DN.FE.ExecuteQuery/8b05892c-23bc-47d2-aec0-9c031e80fc1b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[18:43:33.215]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:43:29.9677918Z);\nlet _endTime = datetime(2026-08-24T18:43:29.9677923Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-76lkckgjnhnb'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:43:33.1912041Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;f1a1de98-d02f-4676-8915-b15cadc2c684\",\r\n \"activityId\": \"5e4cd7bf-9461-4d67-88c2-7821f091c9bf\",\r\n \"subActivityId\": \"d89873a5-1709-4160-8eb3-30bd79fa6a48\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f78e16b5-4442-42a5-8d91-e127bd8a51c3\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;f1a1de98-d02f-4676-8915-b15cadc2c684 ARID=5e4cd7bf-9461-4d67-88c2-7821f091c9bf \u003e GW.Http.CallContext/5e4cd7bf-9461-4d67-88c2-7821f091c9bf \u003e GW.ExecuteQuery/f78e16b5-4442-42a5-8d91-e127bd8a51c3 \u003e DN.FE.ExecuteQuery/d89873a5-1709-4160-8eb3-30bd79fa6a48)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:43:33.267]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:43:29.9677918Z);\nlet _endTime = datetime(2026-08-24T18:43:29.9677923Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-76lkckgjnhnb'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:43:33.1912041Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;f1a1de98-d02f-4676-8915-b15cadc2c684\",\r\n \"activityId\": \"5e4cd7bf-9461-4d67-88c2-7821f091c9bf\",\r\n \"subActivityId\": \"d89873a5-1709-4160-8eb3-30bd79fa6a48\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f78e16b5-4442-42a5-8d91-e127bd8a51c3\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;f1a1de98-d02f-4676-8915-b15cadc2c684 ARID=5e4cd7bf-9461-4d67-88c2-7821f091c9bf \u003e GW.Http.CallContext/5e4cd7bf-9461-4d67-88c2-7821f091c9bf \u003e GW.ExecuteQuery/f78e16b5-4442-42a5-8d91-e127bd8a51c3 \u003e DN.FE.ExecuteQuery/d89873a5-1709-4160-8eb3-30bd79fa6a48)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[18:43:39.952]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:43:36.6153971Z);\nlet _endTime = datetime(2026-08-24T18:43:36.6153978Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-76lkckgjnhnb'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:43:39.9321378Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;4f80dd2a-0964-46f4-a66c-ec4414df5f49\",\r\n \"activityId\": \"e32b2723-acb7-4237-9317-38e8576a7c3c\",\r\n \"subActivityId\": \"7c297192-bc29-4dcc-bd6d-6334b4e363c0\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"e6a1b5c8-345f-4f20-958a-4b363e13136e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;4f80dd2a-0964-46f4-a66c-ec4414df5f49 ARID=e32b2723-acb7-4237-9317-38e8576a7c3c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/e1b634d9-4b0f-48ff-bd25-6198c34b9476 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/e1b634d9-4b0f-48ff-bd25-6198c34b9476 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/e6a1b5c8-345f-4f20-958a-4b363e13136e \u003e DN.FE.ExecuteQuery/7c297192-bc29-4dcc-bd6d-6334b4e363c0)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:43:40.060]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:43:36.6153971Z);\nlet _endTime = datetime(2026-08-24T18:43:36.6153978Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-76lkckgjnhnb'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:43:39.9321378Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;4f80dd2a-0964-46f4-a66c-ec4414df5f49\",\r\n \"activityId\": \"e32b2723-acb7-4237-9317-38e8576a7c3c\",\r\n \"subActivityId\": \"7c297192-bc29-4dcc-bd6d-6334b4e363c0\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"e6a1b5c8-345f-4f20-958a-4b363e13136e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;4f80dd2a-0964-46f4-a66c-ec4414df5f49 ARID=e32b2723-acb7-4237-9317-38e8576a7c3c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/e1b634d9-4b0f-48ff-bd25-6198c34b9476 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/e1b634d9-4b0f-48ff-bd25-6198c34b9476 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/e6a1b5c8-345f-4f20-958a-4b363e13136e \u003e DN.FE.ExecuteQuery/7c297192-bc29-4dcc-bd6d-6334b4e363c0)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.009s. Expected success, but got an error: <errors.errorString \| 0xc001b003c0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[18:43:29.806]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:43:26.1018327Z);\nlet _endTime = datetime(2026-08-24T18:43:26.1018334Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-76lkckgjnhnb'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:43:29.7846996Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;9c4baa00-2868-44b4-9825-b777d40ca5d5\",\r\n \"activityId\": \"7aaddfd7-cd62-422f-9ff8-20bc627c5167\",\r\n \"subActivityId\": \"8b05892c-23bc-47d2-aec0-9c031e80fc1b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"214720a0-3b18-41d3-91a6-6d3d1da6e215\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;9c4baa00-2868-44b4-9825-b777d40ca5d5 ARID=7aaddfd7-cd62-422f-9ff8-20bc627c5167 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/2e60780d-ba60-40ae-9636-d9c2055ea7e4 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/2e60780d-ba60-40ae-9636-d9c2055ea7e4 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/214720a0-3b18-41d3-91a6-6d3d1da6e215 \u003e DN.FE.ExecuteQuery/8b05892c-23bc-47d2-aec0-9c031e80fc1b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:43:29.857]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:43:26.1018327Z);\nlet _endTime = datetime(2026-08-24T18:43:26.1018334Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-76lkckgjnhnb'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:43:29.7846996Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;9c4baa00-2868-44b4-9825-b777d40ca5d5\",\r\n \"activityId\": \"7aaddfd7-cd62-422f-9ff8-20bc627c5167\",\r\n \"subActivityId\": \"8b05892c-23bc-47d2-aec0-9c031e80fc1b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"214720a0-3b18-41d3-91a6-6d3d1da6e215\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;9c4baa00-2868-44b4-9825-b777d40ca5d5 ARID=7aaddfd7-cd62-422f-9ff8-20bc627c5167 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/2e60780d-ba60-40ae-9636-d9c2055ea7e4 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/2e60780d-ba60-40ae-9636-d9c2055ea7e4 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/214720a0-3b18-41d3-91a6-6d3d1da6e215 \u003e DN.FE.ExecuteQuery/8b05892c-23bc-47d2-aec0-9c031e80fc1b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[18:43:33.215]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:43:29.9677918Z);\nlet _endTime = datetime(2026-08-24T18:43:29.9677923Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-76lkckgjnhnb'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:43:33.1912041Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;f1a1de98-d02f-4676-8915-b15cadc2c684\",\r\n \"activityId\": \"5e4cd7bf-9461-4d67-88c2-7821f091c9bf\",\r\n \"subActivityId\": \"d89873a5-1709-4160-8eb3-30bd79fa6a48\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f78e16b5-4442-42a5-8d91-e127bd8a51c3\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;f1a1de98-d02f-4676-8915-b15cadc2c684 ARID=5e4cd7bf-9461-4d67-88c2-7821f091c9bf \u003e GW.Http.CallContext/5e4cd7bf-9461-4d67-88c2-7821f091c9bf \u003e GW.ExecuteQuery/f78e16b5-4442-42a5-8d91-e127bd8a51c3 \u003e DN.FE.ExecuteQuery/d89873a5-1709-4160-8eb3-30bd79fa6a48)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:43:33.267]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:43:29.9677918Z);\nlet _endTime = datetime(2026-08-24T18:43:29.9677923Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-76lkckgjnhnb'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:43:33.1912041Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;f1a1de98-d02f-4676-8915-b15cadc2c684\",\r\n \"activityId\": \"5e4cd7bf-9461-4d67-88c2-7821f091c9bf\",\r\n \"subActivityId\": \"d89873a5-1709-4160-8eb3-30bd79fa6a48\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f78e16b5-4442-42a5-8d91-e127bd8a51c3\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;f1a1de98-d02f-4676-8915-b15cadc2c684 ARID=5e4cd7bf-9461-4d67-88c2-7821f091c9bf \u003e GW.Http.CallContext/5e4cd7bf-9461-4d67-88c2-7821f091c9bf \u003e GW.ExecuteQuery/f78e16b5-4442-42a5-8d91-e127bd8a51c3 \u003e DN.FE.ExecuteQuery/d89873a5-1709-4160-8eb3-30bd79fa6a48)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[18:43:39.952]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:43:36.6153971Z);\nlet _endTime = datetime(2026-08-24T18:43:36.6153978Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-76lkckgjnhnb'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:43:39.9321378Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;4f80dd2a-0964-46f4-a66c-ec4414df5f49\",\r\n \"activityId\": \"e32b2723-acb7-4237-9317-38e8576a7c3c\",\r\n \"subActivityId\": \"7c297192-bc29-4dcc-bd6d-6334b4e363c0\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"e6a1b5c8-345f-4f20-958a-4b363e13136e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;4f80dd2a-0964-46f4-a66c-ec4414df5f49 ARID=e32b2723-acb7-4237-9317-38e8576a7c3c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/e1b634d9-4b0f-48ff-bd25-6198c34b9476 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/e1b634d9-4b0f-48ff-bd25-6198c34b9476 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/e6a1b5c8-345f-4f20-958a-4b363e13136e \u003e DN.FE.ExecuteQuery/7c297192-bc29-4dcc-bd6d-6334b4e363c0)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:43:40.060]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:43:36.6153971Z);\nlet _endTime = datetime(2026-08-24T18:43:36.6153978Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-76lkckgjnhnb'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:43:39.9321378Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;4f80dd2a-0964-46f4-a66c-ec4414df5f49\",\r\n \"activityId\": \"e32b2723-acb7-4237-9317-38e8576a7c3c\",\r\n \"subActivityId\": \"7c297192-bc29-4dcc-bd6d-6334b4e363c0\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"e6a1b5c8-345f-4f20-958a-4b363e13136e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;4f80dd2a-0964-46f4-a66c-ec4414df5f49 ARID=e32b2723-acb7-4237-9317-38e8576a7c3c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/e1b634d9-4b0f-48ff-bd25-6198c34b9476 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/e1b634d9-4b0f-48ff-bd25-6198c34b9476 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/e6a1b5c8-345f-4f20-958a-4b363e13136e \u003e DN.FE.ExecuteQuery/7c297192-bc29-4dcc-bd6d-6334b4e363c0)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ...
17:22:18	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 8ce3ed6	alert, e2e	failed	★#5958 (open)	4	e2e (3) failed to select node pool <nodepool> version for <ocp-channel> channel: Get "<url>": dial tcp: lookup api 17:22:18 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:207]: failed to select node pool install version for candidate-4.20 channel: Get "https://api.openshift.com/api/upgrades_info/graph?arch=multi&channel=candidate-4.20": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.208.156:42093->172.30.0.10:53: i/o timeout fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:207]: failed to select node pool install version for candidate-4.20 channel: Get "https://api.openshift.com/api/upgrades_info/graph?arch=multi&channel=candidate-4.20": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.208.156:42093->172.30.0.10:53: i/o timeout ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 17:22:18 UTC · failed at e2e · test FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc00095a160>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-knxnfrhmdzr6" to finish creating: GET https://rp.j8858240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/03cb2605-91e0-443c-a252-ceeedea9d18a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/03cb2605-91e0-443c-a252-ceeedea9d18a", "name": "03cb2605-91e0-443c-a252-ceeedea9d18a", "status": "Failed", "startTime": "2026-08-24T18:16:27.708109185Z", "endTime": "2026-08-24T18:35:33.013328294Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc00095a160>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-knxnfrhmdzr6" to finish creating: GET https://rp.j8858240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/03cb2605-91e0-443c-a252-ceeedea9d18a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/03cb2605-91e0-443c-a252-ceeedea9d18a", "name": "03cb2605-91e0-443c-a252-ceeedea9d18a", "status": "Failed", "startTime": "2026-08-24T18:16:27.708109185Z", "endTime": "2026-08-24T18:35:33.013328294Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred query-basic: command failed: exit status 1 17:22:18 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.009s. Expected success, but got an error: <errors.errorString \| 0xc000af8160>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[18:38:30.078]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:38:19.1391016Z);\nlet _endTime = datetime(2026-08-24T18:38:19.1391021Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lrs5cwxggm6w'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:38:29.9359819Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;38d751d6-5b8d-4f16-bc3b-6be2306ac850\",\r\n \"activityId\": \"767d7e36-4234-4eec-a531-c0201506201f\",\r\n \"subActivityId\": \"c8289c24-c101-48b8-a652-550358625e95\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"0333b671-454e-490f-8c0f-0844fe7df9fb\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;38d751d6-5b8d-4f16-bc3b-6be2306ac850 ARID=767d7e36-4234-4eec-a531-c0201506201f \u003e GW.Http.CallContext/767d7e36-4234-4eec-a531-c0201506201f \u003e GW.ExecuteQuery/0333b671-454e-490f-8c0f-0844fe7df9fb \u003e DN.FE.ExecuteQuery/c8289c24-c101-48b8-a652-550358625e95)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:38:30.386]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:38:19.1391016Z);\nlet _endTime = datetime(2026-08-24T18:38:19.1391021Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lrs5cwxggm6w'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:38:29.9359819Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;38d751d6-5b8d-4f16-bc3b-6be2306ac850\",\r\n \"activityId\": \"767d7e36-4234-4eec-a531-c0201506201f\",\r\n \"subActivityId\": \"c8289c24-c101-48b8-a652-550358625e95\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"0333b671-454e-490f-8c0f-0844fe7df9fb\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;38d751d6-5b8d-4f16-bc3b-6be2306ac850 ARID=767d7e36-4234-4eec-a531-c0201506201f \u003e GW.Http.CallContext/767d7e36-4234-4eec-a531-c0201506201f \u003e GW.ExecuteQuery/0333b671-454e-490f-8c0f-0844fe7df9fb \u003e DN.FE.ExecuteQuery/c8289c24-c101-48b8-a652-550358625e95)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[18:38:41.465]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:38:30.4518739Z);\nlet _endTime = datetime(2026-08-24T18:38:30.4518744Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lrs5cwxggm6w'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:38:41.4379865Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;23368c59-3298-486f-b16b-51120c31d56d\",\r\n \"activityId\": \"adbfc21a-3f67-4173-9cab-6bf8f9eb3ff8\",\r\n \"subActivityId\": \"7c76994e-b5dc-4137-8df0-91160aa33d94\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"35086d4b-fc80-4580-8f06-30c8a6eab682\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;23368c59-3298-486f-b16b-51120c31d56d ARID=adbfc21a-3f67-4173-9cab-6bf8f9eb3ff8 \u003e GW.Http.CallContext/adbfc21a-3f67-4173-9cab-6bf8f9eb3ff8 \u003e GW.ExecuteQuery/35086d4b-fc80-4580-8f06-30c8a6eab682 \u003e DN.FE.ExecuteQuery/7c76994e-b5dc-4137-8df0-91160aa33d94)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:38:41.522]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:38:30.4518739Z);\nlet _endTime = datetime(2026-08-24T18:38:30.4518744Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lrs5cwxggm6w'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:38:41.4379865Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;23368c59-3298-486f-b16b-51120c31d56d\",\r\n \"activityId\": \"adbfc21a-3f67-4173-9cab-6bf8f9eb3ff8\",\r\n \"subActivityId\": \"7c76994e-b5dc-4137-8df0-91160aa33d94\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"35086d4b-fc80-4580-8f06-30c8a6eab682\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;23368c59-3298-486f-b16b-51120c31d56d ARID=adbfc21a-3f67-4173-9cab-6bf8f9eb3ff8 \u003e GW.Http.CallContext/adbfc21a-3f67-4173-9cab-6bf8f9eb3ff8 \u003e GW.ExecuteQuery/35086d4b-fc80-4580-8f06-30c8a6eab682 \u003e DN.FE.ExecuteQuery/7c76994e-b5dc-4137-8df0-91160aa33d94)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[18:39:12.706]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:38:55.3973069Z);\nlet _endTime = datetime(2026-08-24T18:38:55.3973074Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lrs5cwxggm6w'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:39:12.6828025Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;9d8bb29c-908a-4cdf-90e4-8935c99a813c\",\r\n \"activityId\": \"726ff655-0d5c-46b8-8137-e2ff94407940\",\r\n \"subActivityId\": \"6b3a4ccd-ce64-4c47-8cae-0bdb0403bb3c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"dc7d5fbb-34c4-4834-97c2-f484d996fa95\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;9d8bb29c-908a-4cdf-90e4-8935c99a813c ARID=726ff655-0d5c-46b8-8137-e2ff94407940 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f19dfb92-f849-40a9-955b-3947ba99aa46 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f19dfb92-f849-40a9-955b-3947ba99aa46 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/dc7d5fbb-34c4-4834-97c2-f484d996fa95 \u003e DN.FE.ExecuteQuery/6b3a4ccd-ce64-4c47-8cae-0bdb0403bb3c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:39:13.039]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:38:55.3973069Z);\nlet _endTime = datetime(2026-08-24T18:38:55.3973074Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lrs5cwxggm6w'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:39:12.6828025Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;9d8bb29c-908a-4cdf-90e4-8935c99a813c\",\r\n \"activityId\": \"726ff655-0d5c-46b8-8137-e2ff94407940\",\r\n \"subActivityId\": \"6b3a4ccd-ce64-4c47-8cae-0bdb0403bb3c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"dc7d5fbb-34c4-4834-97c2-f484d996fa95\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;9d8bb29c-908a-4cdf-90e4-8935c99a813c ARID=726ff655-0d5c-46b8-8137-e2ff94407940 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f19dfb92-f849-40a9-955b-3947ba99aa46 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f19dfb92-f849-40a9-955b-3947ba99aa46 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/dc7d5fbb-34c4-4834-97c2-f484d996fa95 \u003e DN.FE.ExecuteQuery/6b3a4ccd-ce64-4c47-8cae-0bdb0403bb3c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.009s. Expected success, but got an error: <errors.errorString \| 0xc000af8160>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[18:38:30.078]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:38:19.1391016Z);\nlet _endTime = datetime(2026-08-24T18:38:19.1391021Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lrs5cwxggm6w'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:38:29.9359819Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;38d751d6-5b8d-4f16-bc3b-6be2306ac850\",\r\n \"activityId\": \"767d7e36-4234-4eec-a531-c0201506201f\",\r\n \"subActivityId\": \"c8289c24-c101-48b8-a652-550358625e95\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"0333b671-454e-490f-8c0f-0844fe7df9fb\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;38d751d6-5b8d-4f16-bc3b-6be2306ac850 ARID=767d7e36-4234-4eec-a531-c0201506201f \u003e GW.Http.CallContext/767d7e36-4234-4eec-a531-c0201506201f \u003e GW.ExecuteQuery/0333b671-454e-490f-8c0f-0844fe7df9fb \u003e DN.FE.ExecuteQuery/c8289c24-c101-48b8-a652-550358625e95)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:38:30.386]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:38:19.1391016Z);\nlet _endTime = datetime(2026-08-24T18:38:19.1391021Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lrs5cwxggm6w'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:38:29.9359819Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;38d751d6-5b8d-4f16-bc3b-6be2306ac850\",\r\n \"activityId\": \"767d7e36-4234-4eec-a531-c0201506201f\",\r\n \"subActivityId\": \"c8289c24-c101-48b8-a652-550358625e95\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"0333b671-454e-490f-8c0f-0844fe7df9fb\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;38d751d6-5b8d-4f16-bc3b-6be2306ac850 ARID=767d7e36-4234-4eec-a531-c0201506201f \u003e GW.Http.CallContext/767d7e36-4234-4eec-a531-c0201506201f \u003e GW.ExecuteQuery/0333b671-454e-490f-8c0f-0844fe7df9fb \u003e DN.FE.ExecuteQuery/c8289c24-c101-48b8-a652-550358625e95)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[18:38:41.465]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:38:30.4518739Z);\nlet _endTime = datetime(2026-08-24T18:38:30.4518744Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lrs5cwxggm6w'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:38:41.4379865Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;23368c59-3298-486f-b16b-51120c31d56d\",\r\n \"activityId\": \"adbfc21a-3f67-4173-9cab-6bf8f9eb3ff8\",\r\n \"subActivityId\": \"7c76994e-b5dc-4137-8df0-91160aa33d94\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"35086d4b-fc80-4580-8f06-30c8a6eab682\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;23368c59-3298-486f-b16b-51120c31d56d ARID=adbfc21a-3f67-4173-9cab-6bf8f9eb3ff8 \u003e GW.Http.CallContext/adbfc21a-3f67-4173-9cab-6bf8f9eb3ff8 \u003e GW.ExecuteQuery/35086d4b-fc80-4580-8f06-30c8a6eab682 \u003e DN.FE.ExecuteQuery/7c76994e-b5dc-4137-8df0-91160aa33d94)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:38:41.522]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:38:30.4518739Z);\nlet _endTime = datetime(2026-08-24T18:38:30.4518744Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lrs5cwxggm6w'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:38:41.4379865Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;23368c59-3298-486f-b16b-51120c31d56d\",\r\n \"activityId\": \"adbfc21a-3f67-4173-9cab-6bf8f9eb3ff8\",\r\n \"subActivityId\": \"7c76994e-b5dc-4137-8df0-91160aa33d94\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"35086d4b-fc80-4580-8f06-30c8a6eab682\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;23368c59-3298-486f-b16b-51120c31d56d ARID=adbfc21a-3f67-4173-9cab-6bf8f9eb3ff8 \u003e GW.Http.CallContext/adbfc21a-3f67-4173-9cab-6bf8f9eb3ff8 \u003e GW.ExecuteQuery/35086d4b-fc80-4580-8f06-30c8a6eab682 \u003e DN.FE.ExecuteQuery/7c76994e-b5dc-4137-8df0-91160aa33d94)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[18:39:12.706]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:38:55.3973069Z);\nlet _endTime = datetime(2026-08-24T18:38:55.3973074Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lrs5cwxggm6w'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:39:12.6828025Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;9d8bb29c-908a-4cdf-90e4-8935c99a813c\",\r\n \"activityId\": \"726ff655-0d5c-46b8-8137-e2ff94407940\",\r\n \"subActivityId\": \"6b3a4ccd-ce64-4c47-8cae-0bdb0403bb3c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"dc7d5fbb-34c4-4834-97c2-f484d996fa95\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;9d8bb29c-908a-4cdf-90e4-8935c99a813c ARID=726ff655-0d5c-46b8-8137-e2ff94407940 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f19dfb92-f849-40a9-955b-3947ba99aa46 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f19dfb92-f849-40a9-955b-3947ba99aa46 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/dc7d5fbb-34c4-4834-97c2-f484d996fa95 \u003e DN.FE.ExecuteQuery/6b3a4ccd-ce64-4c47-8cae-0bdb0403bb3c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:39:13.039]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:38:55.3973069Z);\nlet _endTime = datetime(2026-08-24T18:38:55.3973074Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lrs5cwxggm6w'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:39:12.6828025Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;9d8bb29c-908a-4cdf-90e4-8935c99a813c\",\r\n \"activityId\": \"726ff655-0d5c-46b8-8137-e2ff94407940\",\r\n \"subActivityId\": \"6b3a4ccd-ce64-4c47-8cae-0bdb0403bb3c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"dc7d5fbb-34c4-4834-97c2-f484d996fa95\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;9d8bb29c-908a-4cdf-90e4-8935c99a813c ARID=726ff655-0d5c-46b8-8137-e2ff94407940 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f19dfb92-f849-40a9-955b-3947ba99aa46 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f19dfb92-f849-40a9-955b-3947ba99aa46 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/dc7d5fbb-34c4-4834-97c2-f484d996fa95 \u003e DN.FE.ExecuteQuery/6b3a4ccd-ce64-4c47-8cae-0bdb0403bb3c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... alert (1) alert [hcp] KubePodNotReady fired 17:22:18 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 7 time(s) (1 unknown, 6 known) Firing 1: State: Resolved Started: 2026-08-24T18:29:40Z Ended: 2026-08-24T18:48:41Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8858240-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd3ove2kcm8cumvp5a7nli0cn48b8df-f7o4e8w4g1n8d1y", pod="router-756bb97467-6q2rw", severity="warning" Description: Pod ocm-arohcpci01-2sd3ove2kcm8cumvp5a7nli0cn48b8df-f7o4e8w4g1n8d1y/router-756bb97467-6q2rw has been in a non-ready state for longer than 5 minutes.
16:54:39	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c6d4678	e2e	failed	★#6629 (open)	2	e2e (2) query-basic: command failed: exit status 1 16:54:39 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.001s. Expected success, but got an error: <errors.errorString \| 0xc001aa20b0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[18:10:24.297]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:10:20.4586249Z);\nlet _endTime = datetime(2026-08-24T18:10:20.4586253Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-52n7z2mfccvl'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:10:24.2714273Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;12b2ecfe-09e1-4838-993b-5dcf60d0b406\",\r\n \"activityId\": \"04908727-e2e1-425c-ac4a-03f0802e4426\",\r\n \"subActivityId\": \"c2938410-8cf8-48ad-984d-fb727921dbd7\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"6ec85876-5223-4015-aeb3-32640b5a0ad9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;12b2ecfe-09e1-4838-993b-5dcf60d0b406 ARID=04908727-e2e1-425c-ac4a-03f0802e4426 \u003e GW.Http.CallContext/04908727-e2e1-425c-ac4a-03f0802e4426 \u003e GW.ExecuteQuery/6ec85876-5223-4015-aeb3-32640b5a0ad9 \u003e DN.FE.ExecuteQuery/c2938410-8cf8-48ad-984d-fb727921dbd7)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:10:24.356]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:10:20.4586249Z);\nlet _endTime = datetime(2026-08-24T18:10:20.4586253Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-52n7z2mfccvl'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:10:24.2714273Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;12b2ecfe-09e1-4838-993b-5dcf60d0b406\",\r\n \"activityId\": \"04908727-e2e1-425c-ac4a-03f0802e4426\",\r\n \"subActivityId\": \"c2938410-8cf8-48ad-984d-fb727921dbd7\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"6ec85876-5223-4015-aeb3-32640b5a0ad9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;12b2ecfe-09e1-4838-993b-5dcf60d0b406 ARID=04908727-e2e1-425c-ac4a-03f0802e4426 \u003e GW.Http.CallContext/04908727-e2e1-425c-ac4a-03f0802e4426 \u003e GW.ExecuteQuery/6ec85876-5223-4015-aeb3-32640b5a0ad9 \u003e DN.FE.ExecuteQuery/c2938410-8cf8-48ad-984d-fb727921dbd7)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[18:10:27.782]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:10:24.4180884Z);\nlet _endTime = datetime(2026-08-24T18:10:24.4180888Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-52n7z2mfccvl'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:10:27.7556057Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;5c5731ff-386a-4c60-a4ae-7e8099e48b63\",\r\n \"activityId\": \"ccca268d-b5b0-44b7-bb2c-53f62ce07692\",\r\n \"subActivityId\": \"929136dd-2380-48b4-9972-3d2d6b49e1e2\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"08a8bbb1-cd7f-44af-aa81-2a8d989464b8\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;5c5731ff-386a-4c60-a4ae-7e8099e48b63 ARID=ccca268d-b5b0-44b7-bb2c-53f62ce07692 \u003e GW.Http.CallContext/ccca268d-b5b0-44b7-bb2c-53f62ce07692 \u003e GW.ExecuteQuery/08a8bbb1-cd7f-44af-aa81-2a8d989464b8 \u003e DN.FE.ExecuteQuery/929136dd-2380-48b4-9972-3d2d6b49e1e2)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:10:27.872]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:10:24.4180884Z);\nlet _endTime = datetime(2026-08-24T18:10:24.4180888Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-52n7z2mfccvl'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:10:27.7556057Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;5c5731ff-386a-4c60-a4ae-7e8099e48b63\",\r\n \"activityId\": \"ccca268d-b5b0-44b7-bb2c-53f62ce07692\",\r\n \"subActivityId\": \"929136dd-2380-48b4-9972-3d2d6b49e1e2\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"08a8bbb1-cd7f-44af-aa81-2a8d989464b8\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;5c5731ff-386a-4c60-a4ae-7e8099e48b63 ARID=ccca268d-b5b0-44b7-bb2c-53f62ce07692 \u003e GW.Http.CallContext/ccca268d-b5b0-44b7-bb2c-53f62ce07692 \u003e GW.ExecuteQuery/08a8bbb1-cd7f-44af-aa81-2a8d989464b8 \u003e DN.FE.ExecuteQuery/929136dd-2380-48b4-9972-3d2d6b49e1e2)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[18:10:35.410]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:10:31.5845546Z);\nlet _endTime = datetime(2026-08-24T18:10:31.5845551Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-52n7z2mfccvl'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:10:35.3841956Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;5bfc3306-10af-4dad-9510-c14ce5bf3457\",\r\n \"activityId\": \"2664cc8d-1c67-4701-a9ad-6ccfbc1df942\",\r\n \"subActivityId\": \"573c0901-7d69-4aa6-8cc9-171e30466772\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"c7f178d3-71a1-44c2-a3cb-331e0e92f5fb\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;5bfc3306-10af-4dad-9510-c14ce5bf3457 ARID=2664cc8d-1c67-4701-a9ad-6ccfbc1df942 \u003e GW.Http.CallContext/2664cc8d-1c67-4701-a9ad-6ccfbc1df942 \u003e GW.ExecuteQuery/c7f178d3-71a1-44c2-a3cb-331e0e92f5fb \u003e DN.FE.ExecuteQuery/573c0901-7d69-4aa6-8cc9-171e30466772)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:10:35.526]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:10:31.5845546Z);\nlet _endTime = datetime(2026-08-24T18:10:31.5845551Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-52n7z2mfccvl'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:10:35.3841956Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;5bfc3306-10af-4dad-9510-c14ce5bf3457\",\r\n \"activityId\": \"2664cc8d-1c67-4701-a9ad-6ccfbc1df942\",\r\n \"subActivityId\": \"573c0901-7d69-4aa6-8cc9-171e30466772\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"c7f178d3-71a1-44c2-a3cb-331e0e92f5fb\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;5bfc3306-10af-4dad-9510-c14ce5bf3457 ARID=2664cc8d-1c67-4701-a9ad-6ccfbc1df942 \u003e GW.Http.CallContext/2664cc8d-1c67-4701-a9ad-6ccfbc1df942 \u003e GW.ExecuteQuery/c7f178d3-71a1-44c2-a3cb-331e0e92f5fb \u003e DN.FE.ExecuteQuery/573c0901-7d69-4aa6-8cc9-171e30466772)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.001s. Expected success, but got an error: <errors.errorString \| 0xc001aa20b0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[18:10:24.297]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:10:20.4586249Z);\nlet _endTime = datetime(2026-08-24T18:10:20.4586253Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-52n7z2mfccvl'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:10:24.2714273Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;12b2ecfe-09e1-4838-993b-5dcf60d0b406\",\r\n \"activityId\": \"04908727-e2e1-425c-ac4a-03f0802e4426\",\r\n \"subActivityId\": \"c2938410-8cf8-48ad-984d-fb727921dbd7\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"6ec85876-5223-4015-aeb3-32640b5a0ad9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;12b2ecfe-09e1-4838-993b-5dcf60d0b406 ARID=04908727-e2e1-425c-ac4a-03f0802e4426 \u003e GW.Http.CallContext/04908727-e2e1-425c-ac4a-03f0802e4426 \u003e GW.ExecuteQuery/6ec85876-5223-4015-aeb3-32640b5a0ad9 \u003e DN.FE.ExecuteQuery/c2938410-8cf8-48ad-984d-fb727921dbd7)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:10:24.356]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:10:20.4586249Z);\nlet _endTime = datetime(2026-08-24T18:10:20.4586253Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-52n7z2mfccvl'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:10:24.2714273Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;12b2ecfe-09e1-4838-993b-5dcf60d0b406\",\r\n \"activityId\": \"04908727-e2e1-425c-ac4a-03f0802e4426\",\r\n \"subActivityId\": \"c2938410-8cf8-48ad-984d-fb727921dbd7\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"6ec85876-5223-4015-aeb3-32640b5a0ad9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;12b2ecfe-09e1-4838-993b-5dcf60d0b406 ARID=04908727-e2e1-425c-ac4a-03f0802e4426 \u003e GW.Http.CallContext/04908727-e2e1-425c-ac4a-03f0802e4426 \u003e GW.ExecuteQuery/6ec85876-5223-4015-aeb3-32640b5a0ad9 \u003e DN.FE.ExecuteQuery/c2938410-8cf8-48ad-984d-fb727921dbd7)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[18:10:27.782]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:10:24.4180884Z);\nlet _endTime = datetime(2026-08-24T18:10:24.4180888Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-52n7z2mfccvl'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:10:27.7556057Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;5c5731ff-386a-4c60-a4ae-7e8099e48b63\",\r\n \"activityId\": \"ccca268d-b5b0-44b7-bb2c-53f62ce07692\",\r\n \"subActivityId\": \"929136dd-2380-48b4-9972-3d2d6b49e1e2\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"08a8bbb1-cd7f-44af-aa81-2a8d989464b8\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;5c5731ff-386a-4c60-a4ae-7e8099e48b63 ARID=ccca268d-b5b0-44b7-bb2c-53f62ce07692 \u003e GW.Http.CallContext/ccca268d-b5b0-44b7-bb2c-53f62ce07692 \u003e GW.ExecuteQuery/08a8bbb1-cd7f-44af-aa81-2a8d989464b8 \u003e DN.FE.ExecuteQuery/929136dd-2380-48b4-9972-3d2d6b49e1e2)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:10:27.872]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:10:24.4180884Z);\nlet _endTime = datetime(2026-08-24T18:10:24.4180888Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-52n7z2mfccvl'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:10:27.7556057Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;5c5731ff-386a-4c60-a4ae-7e8099e48b63\",\r\n \"activityId\": \"ccca268d-b5b0-44b7-bb2c-53f62ce07692\",\r\n \"subActivityId\": \"929136dd-2380-48b4-9972-3d2d6b49e1e2\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"08a8bbb1-cd7f-44af-aa81-2a8d989464b8\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;5c5731ff-386a-4c60-a4ae-7e8099e48b63 ARID=ccca268d-b5b0-44b7-bb2c-53f62ce07692 \u003e GW.Http.CallContext/ccca268d-b5b0-44b7-bb2c-53f62ce07692 \u003e GW.ExecuteQuery/08a8bbb1-cd7f-44af-aa81-2a8d989464b8 \u003e DN.FE.ExecuteQuery/929136dd-2380-48b4-9972-3d2d6b49e1e2)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[18:10:35.410]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:10:31.5845546Z);\nlet _endTime = datetime(2026-08-24T18:10:31.5845551Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-52n7z2mfccvl'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:10:35.3841956Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;5bfc3306-10af-4dad-9510-c14ce5bf3457\",\r\n \"activityId\": \"2664cc8d-1c67-4701-a9ad-6ccfbc1df942\",\r\n \"subActivityId\": \"573c0901-7d69-4aa6-8cc9-171e30466772\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"c7f178d3-71a1-44c2-a3cb-331e0e92f5fb\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;5bfc3306-10af-4dad-9510-c14ce5bf3457 ARID=2664cc8d-1c67-4701-a9ad-6ccfbc1df942 \u003e GW.Http.CallContext/2664cc8d-1c67-4701-a9ad-6ccfbc1df942 \u003e GW.ExecuteQuery/c7f178d3-71a1-44c2-a3cb-331e0e92f5fb \u003e DN.FE.ExecuteQuery/573c0901-7d69-4aa6-8cc9-171e30466772)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:10:35.526]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:10:31.5845546Z);\nlet _endTime = datetime(2026-08-24T18:10:31.5845551Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-52n7z2mfccvl'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:10:35.3841956Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;5bfc3306-10af-4dad-9510-c14ce5bf3457\",\r\n \"activityId\": \"2664cc8d-1c67-4701-a9ad-6ccfbc1df942\",\r\n \"subActivityId\": \"573c0901-7d69-4aa6-8cc9-171e30466772\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"c7f178d3-71a1-44c2-a3cb-331e0e92f5fb\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;5bfc3306-10af-4dad-9510-c14ce5bf3457 ARID=2664cc8d-1c67-4701-a9ad-6ccfbc1df942 \u003e GW.Http.CallContext/2664cc8d-1c67-4701-a9ad-6ccfbc1df942 \u003e GW.ExecuteQuery/c7f178d3-71a1-44c2-a3cb-331e0e92f5fb \u003e DN.FE.ExecuteQuery/573c0901-7d69-4aa6-8cc9-171e30466772)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 16:54:39 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-552s2ntf2vdp/cluster-candidate-4-20-j2m8wl should provision Unexpected error: <fmt.wrapError \| 0xc001678380>: failed to create HCP cluster cluster-candidate-4-20-j2m8wl: failed waiting for cluster="cluster-candidate-4-20-j2m8wl" in resourcegroup="rg-candidate-4-20-552s2ntf2vdp" to finish creating: GET https://rp.j0130816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2542dd96-3399-4c2c-935f-47bf06d32f0d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2542dd96-3399-4c2c-935f-47bf06d32f0d", "name": "2542dd96-3399-4c2c-935f-47bf06d32f0d", "status": "Failed", "startTime": "2026-08-24T17:45:24.52790269Z", "endTime": "2026-08-24T18:04:27.860524772Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-552s2ntf2vdp/cluster-candidate-4-20-j2m8wl should provision Unexpected error: <fmt.wrapError \| 0xc001678380>: failed to create HCP cluster cluster-candidate-4-20-j2m8wl: failed waiting for cluster="cluster-candidate-4-20-j2m8wl" in resourcegroup="rg-candidate-4-20-552s2ntf2vdp" to finish creating: GET https://rp.j0130816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2542dd96-3399-4c2c-935f-47bf06d32f0d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2542dd96-3399-4c2c-935f-47bf06d32f0d", "name": "2542dd96-3399-4c2c-935f-47bf06d32f0d", "status": "Failed", "startTime": "2026-08-24T17:45:24.52790269Z", "endTime": "2026-08-24T18:04:27.860524772Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred
16:54:34	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 9906d38	alert, e2e	failed	★#6658 (open)	2	e2e (1) query-basic: command failed: exit status 1 16:54:34 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 606.493s. Expected success, but got an error: <errors.errorString \| 0xc00110c300>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[18:17:21.068]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:17:17.9098588Z);\nlet _endTime = datetime(2026-08-24T18:17:17.9098594Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hw6zxbtrjc8s'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:17:21.0429499Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;50177515-800c-48a2-bc9a-6190f8f64647\",\r\n \"activityId\": \"25584a54-0763-4948-aaed-8fd131374fe3\",\r\n \"subActivityId\": \"83a66406-6dab-4195-b310-e1df353b33ce\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"18a6bb5a-fff3-4c81-8c7a-669d3ce47a8e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;50177515-800c-48a2-bc9a-6190f8f64647 ARID=25584a54-0763-4948-aaed-8fd131374fe3 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/9e4a2718-7c90-4bde-933d-0a3e9ed9c527 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/9e4a2718-7c90-4bde-933d-0a3e9ed9c527 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/18a6bb5a-fff3-4c81-8c7a-669d3ce47a8e \u003e DN.FE.ExecuteQuery/83a66406-6dab-4195-b310-e1df353b33ce)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:17:21.123]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:17:17.9098588Z);\nlet _endTime = datetime(2026-08-24T18:17:17.9098594Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hw6zxbtrjc8s'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:17:21.0429499Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;50177515-800c-48a2-bc9a-6190f8f64647\",\r\n \"activityId\": \"25584a54-0763-4948-aaed-8fd131374fe3\",\r\n \"subActivityId\": \"83a66406-6dab-4195-b310-e1df353b33ce\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"18a6bb5a-fff3-4c81-8c7a-669d3ce47a8e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;50177515-800c-48a2-bc9a-6190f8f64647 ARID=25584a54-0763-4948-aaed-8fd131374fe3 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/9e4a2718-7c90-4bde-933d-0a3e9ed9c527 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/9e4a2718-7c90-4bde-933d-0a3e9ed9c527 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/18a6bb5a-fff3-4c81-8c7a-669d3ce47a8e \u003e DN.FE.ExecuteQuery/83a66406-6dab-4195-b310-e1df353b33ce)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[18:17:24.187]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:17:21.1899023Z);\nlet _endTime = datetime(2026-08-24T18:17:21.1899027Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hw6zxbtrjc8s'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:17:24.1627418Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;deaa39c3-e50e-4c32-9678-afe919848447\",\r\n \"activityId\": \"d854a2b3-b483-42b6-b063-920d81b38162\",\r\n \"subActivityId\": \"92aa21a9-859f-48e1-867d-a294279ddca3\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"c8f9a286-37c0-4d0a-a38e-876553d01a23\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;deaa39c3-e50e-4c32-9678-afe919848447 ARID=d854a2b3-b483-42b6-b063-920d81b38162 \u003e GW.Http.CallContext/d854a2b3-b483-42b6-b063-920d81b38162 \u003e GW.ExecuteQuery/c8f9a286-37c0-4d0a-a38e-876553d01a23 \u003e DN.FE.ExecuteQuery/92aa21a9-859f-48e1-867d-a294279ddca3)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:17:24.238]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:17:21.1899023Z);\nlet _endTime = datetime(2026-08-24T18:17:21.1899027Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hw6zxbtrjc8s'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:17:24.1627418Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;deaa39c3-e50e-4c32-9678-afe919848447\",\r\n \"activityId\": \"d854a2b3-b483-42b6-b063-920d81b38162\",\r\n \"subActivityId\": \"92aa21a9-859f-48e1-867d-a294279ddca3\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"c8f9a286-37c0-4d0a-a38e-876553d01a23\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;deaa39c3-e50e-4c32-9678-afe919848447 ARID=d854a2b3-b483-42b6-b063-920d81b38162 \u003e GW.Http.CallContext/d854a2b3-b483-42b6-b063-920d81b38162 \u003e GW.ExecuteQuery/c8f9a286-37c0-4d0a-a38e-876553d01a23 \u003e DN.FE.ExecuteQuery/92aa21a9-859f-48e1-867d-a294279ddca3)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[18:17:30.943]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:17:27.5661103Z);\nlet _endTime = datetime(2026-08-24T18:17:27.5661107Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hw6zxbtrjc8s'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:17:30.9226160Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;fdeb6cbe-caec-42b0-9433-71e30f446206\",\r\n \"activityId\": \"6b5b443c-b5c8-46e9-a846-0f4ba5b9ed9d\",\r\n \"subActivityId\": \"fe3732e9-d093-4763-911d-1a8ebf675d35\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"2e6c7717-e2dd-4e85-8d2d-85ccd39fd928\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;fdeb6cbe-caec-42b0-9433-71e30f446206 ARID=6b5b443c-b5c8-46e9-a846-0f4ba5b9ed9d \u003e GW.Http.CallContext/6b5b443c-b5c8-46e9-a846-0f4ba5b9ed9d \u003e GW.ExecuteQuery/2e6c7717-e2dd-4e85-8d2d-85ccd39fd928 \u003e DN.FE.ExecuteQuery/fe3732e9-d093-4763-911d-1a8ebf675d35)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:17:31.032]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:17:27.5661103Z);\nlet _endTime = datetime(2026-08-24T18:17:27.5661107Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hw6zxbtrjc8s'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:17:30.9226160Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;fdeb6cbe-caec-42b0-9433-71e30f446206\",\r\n \"activityId\": \"6b5b443c-b5c8-46e9-a846-0f4ba5b9ed9d\",\r\n \"subActivityId\": \"fe3732e9-d093-4763-911d-1a8ebf675d35\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"2e6c7717-e2dd-4e85-8d2d-85ccd39fd928\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;fdeb6cbe-caec-42b0-9433-71e30f446206 ARID=6b5b443c-b5c8-46e9-a846-0f4ba5b9ed9d \u003e GW.Http.CallContext/6b5b443c-b5c8-46e9-a846-0f4ba5b9ed9d \u003e GW.ExecuteQuery/2e6c7717-e2dd-4e85-8d2d-85ccd39fd928 \u003e DN.FE.ExecuteQuery/fe3732e9-d093-4763-911d-1a8ebf675d35)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 606.493s. Expected success, but got an error: <errors.errorString \| 0xc00110c300>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[18:17:21.068]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:17:17.9098588Z);\nlet _endTime = datetime(2026-08-24T18:17:17.9098594Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hw6zxbtrjc8s'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:17:21.0429499Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;50177515-800c-48a2-bc9a-6190f8f64647\",\r\n \"activityId\": \"25584a54-0763-4948-aaed-8fd131374fe3\",\r\n \"subActivityId\": \"83a66406-6dab-4195-b310-e1df353b33ce\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"18a6bb5a-fff3-4c81-8c7a-669d3ce47a8e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;50177515-800c-48a2-bc9a-6190f8f64647 ARID=25584a54-0763-4948-aaed-8fd131374fe3 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/9e4a2718-7c90-4bde-933d-0a3e9ed9c527 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/9e4a2718-7c90-4bde-933d-0a3e9ed9c527 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/18a6bb5a-fff3-4c81-8c7a-669d3ce47a8e \u003e DN.FE.ExecuteQuery/83a66406-6dab-4195-b310-e1df353b33ce)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:17:21.123]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:17:17.9098588Z);\nlet _endTime = datetime(2026-08-24T18:17:17.9098594Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hw6zxbtrjc8s'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:17:21.0429499Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;50177515-800c-48a2-bc9a-6190f8f64647\",\r\n \"activityId\": \"25584a54-0763-4948-aaed-8fd131374fe3\",\r\n \"subActivityId\": \"83a66406-6dab-4195-b310-e1df353b33ce\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"18a6bb5a-fff3-4c81-8c7a-669d3ce47a8e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;50177515-800c-48a2-bc9a-6190f8f64647 ARID=25584a54-0763-4948-aaed-8fd131374fe3 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/9e4a2718-7c90-4bde-933d-0a3e9ed9c527 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/9e4a2718-7c90-4bde-933d-0a3e9ed9c527 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/18a6bb5a-fff3-4c81-8c7a-669d3ce47a8e \u003e DN.FE.ExecuteQuery/83a66406-6dab-4195-b310-e1df353b33ce)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[18:17:24.187]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:17:21.1899023Z);\nlet _endTime = datetime(2026-08-24T18:17:21.1899027Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hw6zxbtrjc8s'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:17:24.1627418Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;deaa39c3-e50e-4c32-9678-afe919848447\",\r\n \"activityId\": \"d854a2b3-b483-42b6-b063-920d81b38162\",\r\n \"subActivityId\": \"92aa21a9-859f-48e1-867d-a294279ddca3\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"c8f9a286-37c0-4d0a-a38e-876553d01a23\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;deaa39c3-e50e-4c32-9678-afe919848447 ARID=d854a2b3-b483-42b6-b063-920d81b38162 \u003e GW.Http.CallContext/d854a2b3-b483-42b6-b063-920d81b38162 \u003e GW.ExecuteQuery/c8f9a286-37c0-4d0a-a38e-876553d01a23 \u003e DN.FE.ExecuteQuery/92aa21a9-859f-48e1-867d-a294279ddca3)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:17:24.238]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:17:21.1899023Z);\nlet _endTime = datetime(2026-08-24T18:17:21.1899027Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hw6zxbtrjc8s'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:17:24.1627418Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;deaa39c3-e50e-4c32-9678-afe919848447\",\r\n \"activityId\": \"d854a2b3-b483-42b6-b063-920d81b38162\",\r\n \"subActivityId\": \"92aa21a9-859f-48e1-867d-a294279ddca3\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"c8f9a286-37c0-4d0a-a38e-876553d01a23\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;deaa39c3-e50e-4c32-9678-afe919848447 ARID=d854a2b3-b483-42b6-b063-920d81b38162 \u003e GW.Http.CallContext/d854a2b3-b483-42b6-b063-920d81b38162 \u003e GW.ExecuteQuery/c8f9a286-37c0-4d0a-a38e-876553d01a23 \u003e DN.FE.ExecuteQuery/92aa21a9-859f-48e1-867d-a294279ddca3)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[18:17:30.943]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:17:27.5661103Z);\nlet _endTime = datetime(2026-08-24T18:17:27.5661107Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hw6zxbtrjc8s'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:17:30.9226160Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;fdeb6cbe-caec-42b0-9433-71e30f446206\",\r\n \"activityId\": \"6b5b443c-b5c8-46e9-a846-0f4ba5b9ed9d\",\r\n \"subActivityId\": \"fe3732e9-d093-4763-911d-1a8ebf675d35\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"2e6c7717-e2dd-4e85-8d2d-85ccd39fd928\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;fdeb6cbe-caec-42b0-9433-71e30f446206 ARID=6b5b443c-b5c8-46e9-a846-0f4ba5b9ed9d \u003e GW.Http.CallContext/6b5b443c-b5c8-46e9-a846-0f4ba5b9ed9d \u003e GW.ExecuteQuery/2e6c7717-e2dd-4e85-8d2d-85ccd39fd928 \u003e DN.FE.ExecuteQuery/fe3732e9-d093-4763-911d-1a8ebf675d35)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:17:31.032]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:17:27.5661103Z);\nlet _endTime = datetime(2026-08-24T18:17:27.5661107Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hw6zxbtrjc8s'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:17:30.9226160Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;fdeb6cbe-caec-42b0-9433-71e30f446206\",\r\n \"activityId\": \"6b5b443c-b5c8-46e9-a846-0f4ba5b9ed9d\",\r\n \"subActivityId\": \"fe3732e9-d093-4763-911d-1a8ebf675d35\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"2e6c7717-e2dd-4e85-8d2d-85ccd39fd928\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;fdeb6cbe-caec-42b0-9433-71e30f446206 ARID=6b5b443c-b5c8-46e9-a846-0f4ba5b9ed9d \u003e GW.Http.CallContext/6b5b443c-b5c8-46e9-a846-0f4ba5b9ed9d \u003e GW.ExecuteQuery/2e6c7717-e2dd-4e85-8d2d-85ccd39fd928 \u003e DN.FE.ExecuteQuery/fe3732e9-d093-4763-911d-1a8ebf675d35)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... alert (1) alert [svc] FrontendPathMedianLatency fired 16:54:34 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-24T17:55:35Z Ended: 2026-08-24T18:27:33Z Severity: Sev3 Labels: alertname="FrontendPathMedianLatency", cluster="ci01-j6361088-svc", component="frontend", method="put", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning" Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j6361088-svc).
16:43:08	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 54bb2db	alert, e2e	failed	★#6575 (open)	6	e2e (2) cluster entered terminal Failed state after role assignment deployment 16:43:08 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.002s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.002s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... query-basic: command failed: exit status 1 16:43:08 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.001s. Expected success, but got an error: <errors.errorString \| 0xc000ead3d0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[18:00:58.615]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:00:50.5032793Z);\nlet _endTime = datetime(2026-08-24T18:00:50.5032799Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-tpxlr6vxklmr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:00:58.5927495Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;960b7b75-5e41-44c9-8108-4f52ccc0d311\",\r\n \"activityId\": \"20001fbf-b892-4112-a9ca-caee90ec5f4d\",\r\n \"subActivityId\": \"b6b4f2e0-1d41-4ffe-bab5-da795807f34f\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"76d1ef04-9778-412c-9b76-1314a76809fb\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;960b7b75-5e41-44c9-8108-4f52ccc0d311 ARID=20001fbf-b892-4112-a9ca-caee90ec5f4d \u003e GW.Http.CallContext/20001fbf-b892-4112-a9ca-caee90ec5f4d \u003e GW.ExecuteQuery/76d1ef04-9778-412c-9b76-1314a76809fb \u003e DN.FE.ExecuteQuery/b6b4f2e0-1d41-4ffe-bab5-da795807f34f)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:00:58.712]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:00:50.5032793Z);\nlet _endTime = datetime(2026-08-24T18:00:50.5032799Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-tpxlr6vxklmr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:00:58.5927495Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;960b7b75-5e41-44c9-8108-4f52ccc0d311\",\r\n \"activityId\": \"20001fbf-b892-4112-a9ca-caee90ec5f4d\",\r\n \"subActivityId\": \"b6b4f2e0-1d41-4ffe-bab5-da795807f34f\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"76d1ef04-9778-412c-9b76-1314a76809fb\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;960b7b75-5e41-44c9-8108-4f52ccc0d311 ARID=20001fbf-b892-4112-a9ca-caee90ec5f4d \u003e GW.Http.CallContext/20001fbf-b892-4112-a9ca-caee90ec5f4d \u003e GW.ExecuteQuery/76d1ef04-9778-412c-9b76-1314a76809fb \u003e DN.FE.ExecuteQuery/b6b4f2e0-1d41-4ffe-bab5-da795807f34f)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[18:01:06.125]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:00:58.8699941Z);\nlet _endTime = datetime(2026-08-24T18:00:58.8699948Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-tpxlr6vxklmr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:01:06.0919619Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;b7b85d2a-f498-4e2b-ba1d-88839731fa27\",\r\n \"activityId\": \"56a132e1-8a08-4840-b33d-04f01dd6e5d3\",\r\n \"subActivityId\": \"dec8312a-b74e-46c5-b158-9db7506079c3\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"53d73257-f47b-435c-89d6-f90c4b661777\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;b7b85d2a-f498-4e2b-ba1d-88839731fa27 ARID=56a132e1-8a08-4840-b33d-04f01dd6e5d3 \u003e GW.Http.CallContext/56a132e1-8a08-4840-b33d-04f01dd6e5d3 \u003e GW.ExecuteQuery/53d73257-f47b-435c-89d6-f90c4b661777 \u003e DN.FE.ExecuteQuery/dec8312a-b74e-46c5-b158-9db7506079c3)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:01:06.626]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:00:58.8699941Z);\nlet _endTime = datetime(2026-08-24T18:00:58.8699948Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-tpxlr6vxklmr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:01:06.0919619Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;b7b85d2a-f498-4e2b-ba1d-88839731fa27\",\r\n \"activityId\": \"56a132e1-8a08-4840-b33d-04f01dd6e5d3\",\r\n \"subActivityId\": \"dec8312a-b74e-46c5-b158-9db7506079c3\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"53d73257-f47b-435c-89d6-f90c4b661777\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;b7b85d2a-f498-4e2b-ba1d-88839731fa27 ARID=56a132e1-8a08-4840-b33d-04f01dd6e5d3 \u003e GW.Http.CallContext/56a132e1-8a08-4840-b33d-04f01dd6e5d3 \u003e GW.ExecuteQuery/53d73257-f47b-435c-89d6-f90c4b661777 \u003e DN.FE.ExecuteQuery/dec8312a-b74e-46c5-b158-9db7506079c3)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[18:01:23.014]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:01:15.4440366Z);\nlet _endTime = datetime(2026-08-24T18:01:15.4440371Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-tpxlr6vxklmr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:01:22.9872186Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;407e8e47-c034-4e41-9520-8819e14f5893\",\r\n \"activityId\": \"5bdc704e-4e63-48a3-9d02-31d081e3ede4\",\r\n \"subActivityId\": \"6e8b566e-338b-412a-ade7-cd2f9e654f08\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"ff01224f-8ad0-44ce-9ce7-3090f402643e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;407e8e47-c034-4e41-9520-8819e14f5893 ARID=5bdc704e-4e63-48a3-9d02-31d081e3ede4 \u003e GW.Http.CallContext/5bdc704e-4e63-48a3-9d02-31d081e3ede4 \u003e GW.ExecuteQuery/ff01224f-8ad0-44ce-9ce7-3090f402643e \u003e DN.FE.ExecuteQuery/6e8b566e-338b-412a-ade7-cd2f9e654f08)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:01:23.290]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:01:15.4440366Z);\nlet _endTime = datetime(2026-08-24T18:01:15.4440371Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-tpxlr6vxklmr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:01:22.9872186Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;407e8e47-c034-4e41-9520-8819e14f5893\",\r\n \"activityId\": \"5bdc704e-4e63-48a3-9d02-31d081e3ede4\",\r\n \"subActivityId\": \"6e8b566e-338b-412a-ade7-cd2f9e654f08\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"ff01224f-8ad0-44ce-9ce7-3090f402643e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;407e8e47-c034-4e41-9520-8819e14f5893 ARID=5bdc704e-4e63-48a3-9d02-31d081e3ede4 \u003e GW.Http.CallContext/5bdc704e-4e63-48a3-9d02-31d081e3ede4 \u003e GW.ExecuteQuery/ff01224f-8ad0-44ce-9ce7-3090f402643e \u003e DN.FE.ExecuteQuery/6e8b566e-338b-412a-ade7-cd2f9e654f08)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.001s. Expected success, but got an error: <errors.errorString \| 0xc000ead3d0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[18:00:58.615]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:00:50.5032793Z);\nlet _endTime = datetime(2026-08-24T18:00:50.5032799Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-tpxlr6vxklmr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:00:58.5927495Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;960b7b75-5e41-44c9-8108-4f52ccc0d311\",\r\n \"activityId\": \"20001fbf-b892-4112-a9ca-caee90ec5f4d\",\r\n \"subActivityId\": \"b6b4f2e0-1d41-4ffe-bab5-da795807f34f\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"76d1ef04-9778-412c-9b76-1314a76809fb\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;960b7b75-5e41-44c9-8108-4f52ccc0d311 ARID=20001fbf-b892-4112-a9ca-caee90ec5f4d \u003e GW.Http.CallContext/20001fbf-b892-4112-a9ca-caee90ec5f4d \u003e GW.ExecuteQuery/76d1ef04-9778-412c-9b76-1314a76809fb \u003e DN.FE.ExecuteQuery/b6b4f2e0-1d41-4ffe-bab5-da795807f34f)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:00:58.712]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:00:50.5032793Z);\nlet _endTime = datetime(2026-08-24T18:00:50.5032799Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-tpxlr6vxklmr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:00:58.5927495Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;960b7b75-5e41-44c9-8108-4f52ccc0d311\",\r\n \"activityId\": \"20001fbf-b892-4112-a9ca-caee90ec5f4d\",\r\n \"subActivityId\": \"b6b4f2e0-1d41-4ffe-bab5-da795807f34f\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"76d1ef04-9778-412c-9b76-1314a76809fb\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;960b7b75-5e41-44c9-8108-4f52ccc0d311 ARID=20001fbf-b892-4112-a9ca-caee90ec5f4d \u003e GW.Http.CallContext/20001fbf-b892-4112-a9ca-caee90ec5f4d \u003e GW.ExecuteQuery/76d1ef04-9778-412c-9b76-1314a76809fb \u003e DN.FE.ExecuteQuery/b6b4f2e0-1d41-4ffe-bab5-da795807f34f)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[18:01:06.125]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:00:58.8699941Z);\nlet _endTime = datetime(2026-08-24T18:00:58.8699948Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-tpxlr6vxklmr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:01:06.0919619Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;b7b85d2a-f498-4e2b-ba1d-88839731fa27\",\r\n \"activityId\": \"56a132e1-8a08-4840-b33d-04f01dd6e5d3\",\r\n \"subActivityId\": \"dec8312a-b74e-46c5-b158-9db7506079c3\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"53d73257-f47b-435c-89d6-f90c4b661777\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;b7b85d2a-f498-4e2b-ba1d-88839731fa27 ARID=56a132e1-8a08-4840-b33d-04f01dd6e5d3 \u003e GW.Http.CallContext/56a132e1-8a08-4840-b33d-04f01dd6e5d3 \u003e GW.ExecuteQuery/53d73257-f47b-435c-89d6-f90c4b661777 \u003e DN.FE.ExecuteQuery/dec8312a-b74e-46c5-b158-9db7506079c3)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:01:06.626]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:00:58.8699941Z);\nlet _endTime = datetime(2026-08-24T18:00:58.8699948Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-tpxlr6vxklmr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:01:06.0919619Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;b7b85d2a-f498-4e2b-ba1d-88839731fa27\",\r\n \"activityId\": \"56a132e1-8a08-4840-b33d-04f01dd6e5d3\",\r\n \"subActivityId\": \"dec8312a-b74e-46c5-b158-9db7506079c3\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"53d73257-f47b-435c-89d6-f90c4b661777\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;b7b85d2a-f498-4e2b-ba1d-88839731fa27 ARID=56a132e1-8a08-4840-b33d-04f01dd6e5d3 \u003e GW.Http.CallContext/56a132e1-8a08-4840-b33d-04f01dd6e5d3 \u003e GW.ExecuteQuery/53d73257-f47b-435c-89d6-f90c4b661777 \u003e DN.FE.ExecuteQuery/dec8312a-b74e-46c5-b158-9db7506079c3)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[18:01:23.014]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:01:15.4440366Z);\nlet _endTime = datetime(2026-08-24T18:01:15.4440371Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-tpxlr6vxklmr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:01:22.9872186Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;407e8e47-c034-4e41-9520-8819e14f5893\",\r\n \"activityId\": \"5bdc704e-4e63-48a3-9d02-31d081e3ede4\",\r\n \"subActivityId\": \"6e8b566e-338b-412a-ade7-cd2f9e654f08\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"ff01224f-8ad0-44ce-9ce7-3090f402643e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;407e8e47-c034-4e41-9520-8819e14f5893 ARID=5bdc704e-4e63-48a3-9d02-31d081e3ede4 \u003e GW.Http.CallContext/5bdc704e-4e63-48a3-9d02-31d081e3ede4 \u003e GW.ExecuteQuery/ff01224f-8ad0-44ce-9ce7-3090f402643e \u003e DN.FE.ExecuteQuery/6e8b566e-338b-412a-ade7-cd2f9e654f08)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:01:23.290]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:01:15.4440366Z);\nlet _endTime = datetime(2026-08-24T18:01:15.4440371Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-tpxlr6vxklmr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:01:22.9872186Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;407e8e47-c034-4e41-9520-8819e14f5893\",\r\n \"activityId\": \"5bdc704e-4e63-48a3-9d02-31d081e3ede4\",\r\n \"subActivityId\": \"6e8b566e-338b-412a-ade7-cd2f9e654f08\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"ff01224f-8ad0-44ce-9ce7-3090f402643e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;407e8e47-c034-4e41-9520-8819e14f5893 ARID=5bdc704e-4e63-48a3-9d02-31d081e3ede4 \u003e GW.Http.CallContext/5bdc704e-4e63-48a3-9d02-31d081e3ede4 \u003e GW.ExecuteQuery/ff01224f-8ad0-44ce-9ce7-3090f402643e \u003e DN.FE.ExecuteQuery/6e8b566e-338b-412a-ade7-cd2f9e654f08)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... alert (4) alert [hcp] KubeDeploymentReplicasMismatch fired 16:43:08 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-24T18:15:21Z Ended: 2026-08-24T18:36:22Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0375808-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.140:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0375808/providers/microsoft.monitor/accounts/hcps-j0375808", namespace="ocm-arohcpci01-2sd37mkptl2qprba4ejkvlbldgl77u9k-r3i6h6d2n5v5b4i", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-hf5qz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2sd37mkptl2qprba4ejkvlbldgl77u9k-r3i6h6d2n5v5b4i/capi-provider has not matched the expected number of replicas for longer than 30 minutes. alert [hcp] KubeStatefulSetReplicasMismatch fired 16:43:08 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-08-24T18:01:25Z Ended: 2026-08-24T18:34:22Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0375808-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.140:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0375808/providers/microsoft.monitor/accounts/hcps-j0375808", namespace="ocm-arohcpci01-2sd37mkptl2qprba4ejkvlbldgl77u9k-r3i6h6d2n5v5b4i", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-hf5qz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2sd37mkptl2qprba4ejkvlbldgl77u9k-r3i6h6d2n5v5b4i/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 2: State: Resolved Started: 2026-08-24T18:01:25Z Ended: 2026-08-24T18:34:22Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0375808-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.140:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0375808/providers/microsoft.monitor/accounts/hcps-j0375808", namespace="ocm-arohcpci01-2sd37mkptl2qprba4ejkvlbldgl77u9k-r3i6h6d2n5v5b4i", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-hf5qz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2sd37mkptl2qprba4ejkvlbldgl77u9k-r3i6h6d2n5v5b4i/etcd has not matched the expected number of replicas for longer than 15 minutes. alert [svc] KubeDeploymentReplicasMismatch fired 16:43:08 UTC · failed at alert · test [aro-hcp-observability] [svc] alert KubeDeploymentReplicasMismatch does not fire · suite aro-hcp-tests Show raw failure alert fired 6 time(s) Firing 1: State: Resolved Started: 2026-08-24T18:15:41Z Ended: 2026-08-24T18:21:45Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0375808-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.140:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0375808/providers/microsoft.monitor/accounts/services-j0375808", namespace="klusterlet-2sd37mkptl2qprba4ejkvlbldgl77u9k", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-hf5qz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2sd37mkptl2qprba4ejkvlbldgl77u9k/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Resolved Started: 2026-08-24T18:15:41Z Ended: 2026-08-24T18:22:41Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0375808-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.140:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0375808/providers/microsoft.monitor/accounts/services-j0375808", namespace="klusterlet-2sd37mkptl2qprba4ejkvlbldgl77u9k", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-hf5qz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2sd37mkptl2qprba4ejkvlbldgl77u9k/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes. Firing 3: State: Resolved Started: 2026-08-24T18:15:41Z Ended: 2026-08-24T18:22:41Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0375808-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.140:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0375808/providers/microsoft.monitor/accounts/services-j0375808", namespace="klusterlet-2sd37mkptl2qprba4ejkvlbldgl77u9k", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-hf5qz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2sd37mkptl2qprba4ejkvlbldgl77u9k/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes. Firing 4: State: Resolved Started: 2026-08-24T18:15:41Z Ended: 2026-08-24T18:23:41Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0375808-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.140:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0375808/providers/microsoft.monitor/accounts/services-j0375808", namespace="klusterlet-2sd37mkptl2qprba4ejkvlbldgl77u9k", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-hf5qz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2sd37mkptl2qprba4ejkvlbldgl77u9k/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes. Firing 5: State: Resolved Started: 2026-08-24T18:15:41Z Ended: 2026-08-24T18:22:41Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0375808-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.140:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0375808/providers/microsoft.monitor/accounts/services-j0375808", namespace="klusterlet-2sd37mkptl2qprba4ejkvlbldgl77u9k", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-hf5qz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2sd37mkptl2qprba4ejkvlbldgl77u9k/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes. Firing 6: State: Resolved Started: 2026-08-24T18:15:41Z Ended: 2026-08-24T18:23:41Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0375808-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.140:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0375808/providers/microsoft.monitor/accounts/services-j0375808", namespace="klusterlet-2sd37mkptl2qprba4ejkvlbldgl77u9k", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-hf5qz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2sd37mkptl2qprba4ejkvlbldgl77u9k/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes. alert [hcp] KubePodCrashLooping fired 16:43:08 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-08-24T18:02:20Z Ended: 2026-08-24T18:38:21Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0375808-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.140:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0375808/providers/microsoft.monitor/accounts/hcps-j0375808", namespace="ocm-arohcpci01-2sd37mkptl2qprba4ejkvlbldgl77u9k-r3i6h6d2n5v5b4i", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="4cebd923-31c4-4c0a-b568-63bf1a281b9a" Description: Pod ocm-arohcpci01-2sd37mkptl2qprba4ejkvlbldgl77u9k-r3i6h6d2n5v5b4i/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Resolved Started: 2026-08-24T18:03:20Z Ended: 2026-08-24T18:38:21Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0375808-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.140:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0375808/providers/microsoft.monitor/accounts/hcps-j0375808", namespace="ocm-arohcpci01-2sd37mkptl2qprba4ejkvlbldgl77u9k-r3i6h6d2n5v5b4i", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="4cebd923-31c4-4c0a-b568-63bf1a281b9a" Description: Pod ocm-arohcpci01-2sd37mkptl2qprba4ejkvlbldgl77u9k-r3i6h6d2n5v5b4i/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
16:40:39	pull-ci-Azure-ARO-HCP-main-e2e-parallel head de4526f	alert, e2e	failed	★#6591 (open)	8	e2e (6) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 16:40:39 UTC · failed at e2e · test SRE can pause schedules to stop backup execution for an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc0019d4200>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-l2vfsmsdwlz6" to finish creating: GET https://rp.j0629888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7fff289f-807c-4680-8e8c-4687765fb3f9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7fff289f-807c-4680-8e8c-4687765fb3f9", "name": "7fff289f-807c-4680-8e8c-4687765fb3f9", "status": "Failed", "startTime": "2026-08-24T17:34:15.71678435Z", "endTime": "2026-08-24T17:53:23.588048645Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc0019d4200>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-l2vfsmsdwlz6" to finish creating: GET https://rp.j0629888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7fff289f-807c-4680-8e8c-4687765fb3f9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7fff289f-807c-4680-8e8c-4687765fb3f9", "name": "7fff289f-807c-4680-8e8c-4687765fb3f9", "status": "Failed", "startTime": "2026-08-24T17:34:15.71678435Z", "endTime": "2026-08-24T17:53:23.588048645Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 16:40:39 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-nzmkwtbzb8xk/cluster-candidate-4-21-k47dlw should provision Unexpected error: <fmt.wrapError \| 0xc00011d300>: failed to create HCP cluster cluster-candidate-4-21-k47dlw: failed waiting for cluster="cluster-candidate-4-21-k47dlw" in resourcegroup="rg-candidate-4-21-nzmkwtbzb8xk" to finish creating: GET https://rp.j0629888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e224492a-2630-4a37-b5e0-7ef00f6cd9dd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e224492a-2630-4a37-b5e0-7ef00f6cd9dd", "name": "e224492a-2630-4a37-b5e0-7ef00f6cd9dd", "status": "Failed", "startTime": "2026-08-24T17:34:09.560283017Z", "endTime": "2026-08-24T17:53:13.775593008Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-nzmkwtbzb8xk/cluster-candidate-4-21-k47dlw should provision Unexpected error: <fmt.wrapError \| 0xc00011d300>: failed to create HCP cluster cluster-candidate-4-21-k47dlw: failed waiting for cluster="cluster-candidate-4-21-k47dlw" in resourcegroup="rg-candidate-4-21-nzmkwtbzb8xk" to finish creating: GET https://rp.j0629888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e224492a-2630-4a37-b5e0-7ef00f6cd9dd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e224492a-2630-4a37-b5e0-7ef00f6cd9dd", "name": "e224492a-2630-4a37-b5e0-7ef00f6cd9dd", "status": "Failed", "startTime": "2026-08-24T17:34:09.560283017Z", "endTime": "2026-08-24T17:53:13.775593008Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred query-basic: command failed: exit status 1 16:40:39 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 602.326s. Expected success, but got an error: <errors.errorString \| 0xc00174c0b0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[18:02:15.191]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:02:09.5899113Z);\nlet _endTime = datetime(2026-08-24T18:02:09.5899117Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hxrz24kgz22q'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:02:15.1649816Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;dc1be810-20f7-4576-9c23-8c49c10db75e\",\r\n \"activityId\": \"8b50e128-6bf1-4c67-b4e1-a93617248240\",\r\n \"subActivityId\": \"d4d65d0b-3d08-4bae-8e97-02e2aa50a413\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"d1dc26ed-fe0d-4b0a-a356-ac3586800c00\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;dc1be810-20f7-4576-9c23-8c49c10db75e ARID=8b50e128-6bf1-4c67-b4e1-a93617248240 \u003e GW.Http.CallContext/8b50e128-6bf1-4c67-b4e1-a93617248240 \u003e GW.ExecuteQuery/d1dc26ed-fe0d-4b0a-a356-ac3586800c00 \u003e DN.FE.ExecuteQuery/d4d65d0b-3d08-4bae-8e97-02e2aa50a413)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:02:15.411]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:02:09.5899113Z);\nlet _endTime = datetime(2026-08-24T18:02:09.5899117Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hxrz24kgz22q'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:02:15.1649816Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;dc1be810-20f7-4576-9c23-8c49c10db75e\",\r\n \"activityId\": \"8b50e128-6bf1-4c67-b4e1-a93617248240\",\r\n \"subActivityId\": \"d4d65d0b-3d08-4bae-8e97-02e2aa50a413\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"d1dc26ed-fe0d-4b0a-a356-ac3586800c00\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;dc1be810-20f7-4576-9c23-8c49c10db75e ARID=8b50e128-6bf1-4c67-b4e1-a93617248240 \u003e GW.Http.CallContext/8b50e128-6bf1-4c67-b4e1-a93617248240 \u003e GW.ExecuteQuery/d1dc26ed-fe0d-4b0a-a356-ac3586800c00 \u003e DN.FE.ExecuteQuery/d4d65d0b-3d08-4bae-8e97-02e2aa50a413)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[18:02:22.018]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:02:15.4920397Z);\nlet _endTime = datetime(2026-08-24T18:02:15.4920402Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hxrz24kgz22q'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:02:21.9924804Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;0d1d7012-9bec-42ca-94eb-41a4c8ac0207\",\r\n \"activityId\": \"7c88679c-59d7-49a4-a8e0-97888f2c387f\",\r\n \"subActivityId\": \"20c41293-ee06-43b0-9f85-1faf9b5ab846\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"0bf6b207-9c8c-43f0-a099-76d33cc8d11d\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;0d1d7012-9bec-42ca-94eb-41a4c8ac0207 ARID=7c88679c-59d7-49a4-a8e0-97888f2c387f \u003e GW.Http.CallContext/7c88679c-59d7-49a4-a8e0-97888f2c387f \u003e GW.ExecuteQuery/0bf6b207-9c8c-43f0-a099-76d33cc8d11d \u003e DN.FE.ExecuteQuery/20c41293-ee06-43b0-9f85-1faf9b5ab846)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:02:22.174]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:02:15.4920397Z);\nlet _endTime = datetime(2026-08-24T18:02:15.4920402Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hxrz24kgz22q'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:02:21.9924804Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;0d1d7012-9bec-42ca-94eb-41a4c8ac0207\",\r\n \"activityId\": \"7c88679c-59d7-49a4-a8e0-97888f2c387f\",\r\n \"subActivityId\": \"20c41293-ee06-43b0-9f85-1faf9b5ab846\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"0bf6b207-9c8c-43f0-a099-76d33cc8d11d\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;0d1d7012-9bec-42ca-94eb-41a4c8ac0207 ARID=7c88679c-59d7-49a4-a8e0-97888f2c387f \u003e GW.Http.CallContext/7c88679c-59d7-49a4-a8e0-97888f2c387f \u003e GW.ExecuteQuery/0bf6b207-9c8c-43f0-a099-76d33cc8d11d \u003e DN.FE.ExecuteQuery/20c41293-ee06-43b0-9f85-1faf9b5ab846)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[18:02:34.313]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:02:27.8203571Z);\nlet _endTime = datetime(2026-08-24T18:02:27.8203574Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hxrz24kgz22q'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:02:34.2861093Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;7b69ab3c-142e-42fc-ba7f-d284b2e71d38\",\r\n \"activityId\": \"833f4c61-46f8-4fdc-895d-d0121d895d64\",\r\n \"subActivityId\": \"18d5bb74-4c1c-4003-b6ff-d5c3cbb43607\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"cbd0d9b4-40b0-49a2-b3c0-ff7e4292d28e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;7b69ab3c-142e-42fc-ba7f-d284b2e71d38 ARID=833f4c61-46f8-4fdc-895d-d0121d895d64 \u003e GW.Http.CallContext/833f4c61-46f8-4fdc-895d-d0121d895d64 \u003e GW.ExecuteQuery/cbd0d9b4-40b0-49a2-b3c0-ff7e4292d28e \u003e DN.FE.ExecuteQuery/18d5bb74-4c1c-4003-b6ff-d5c3cbb43607)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:02:34.447]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:02:27.8203571Z);\nlet _endTime = datetime(2026-08-24T18:02:27.8203574Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hxrz24kgz22q'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:02:34.2861093Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;7b69ab3c-142e-42fc-ba7f-d284b2e71d38\",\r\n \"activityId\": \"833f4c61-46f8-4fdc-895d-d0121d895d64\",\r\n \"subActivityId\": \"18d5bb74-4c1c-4003-b6ff-d5c3cbb43607\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"cbd0d9b4-40b0-49a2-b3c0-ff7e4292d28e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;7b69ab3c-142e-42fc-ba7f-d284b2e71d38 ARID=833f4c61-46f8-4fdc-895d-d0121d895d64 \u003e GW.Http.CallContext/833f4c61-46f8-4fdc-895d-d0121d895d64 \u003e GW.ExecuteQuery/cbd0d9b4-40b0-49a2-b3c0-ff7e4292d28e \u003e DN.FE.ExecuteQuery/18d5bb74-4c1c-4003-b6ff-d5c3cbb43607)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 602.326s. Expected success, but got an error: <errors.errorString \| 0xc00174c0b0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[18:02:15.191]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:02:09.5899113Z);\nlet _endTime = datetime(2026-08-24T18:02:09.5899117Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hxrz24kgz22q'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:02:15.1649816Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;dc1be810-20f7-4576-9c23-8c49c10db75e\",\r\n \"activityId\": \"8b50e128-6bf1-4c67-b4e1-a93617248240\",\r\n \"subActivityId\": \"d4d65d0b-3d08-4bae-8e97-02e2aa50a413\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"d1dc26ed-fe0d-4b0a-a356-ac3586800c00\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;dc1be810-20f7-4576-9c23-8c49c10db75e ARID=8b50e128-6bf1-4c67-b4e1-a93617248240 \u003e GW.Http.CallContext/8b50e128-6bf1-4c67-b4e1-a93617248240 \u003e GW.ExecuteQuery/d1dc26ed-fe0d-4b0a-a356-ac3586800c00 \u003e DN.FE.ExecuteQuery/d4d65d0b-3d08-4bae-8e97-02e2aa50a413)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:02:15.411]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:02:09.5899113Z);\nlet _endTime = datetime(2026-08-24T18:02:09.5899117Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hxrz24kgz22q'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:02:15.1649816Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;dc1be810-20f7-4576-9c23-8c49c10db75e\",\r\n \"activityId\": \"8b50e128-6bf1-4c67-b4e1-a93617248240\",\r\n \"subActivityId\": \"d4d65d0b-3d08-4bae-8e97-02e2aa50a413\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"d1dc26ed-fe0d-4b0a-a356-ac3586800c00\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;dc1be810-20f7-4576-9c23-8c49c10db75e ARID=8b50e128-6bf1-4c67-b4e1-a93617248240 \u003e GW.Http.CallContext/8b50e128-6bf1-4c67-b4e1-a93617248240 \u003e GW.ExecuteQuery/d1dc26ed-fe0d-4b0a-a356-ac3586800c00 \u003e DN.FE.ExecuteQuery/d4d65d0b-3d08-4bae-8e97-02e2aa50a413)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[18:02:22.018]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:02:15.4920397Z);\nlet _endTime = datetime(2026-08-24T18:02:15.4920402Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hxrz24kgz22q'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:02:21.9924804Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;0d1d7012-9bec-42ca-94eb-41a4c8ac0207\",\r\n \"activityId\": \"7c88679c-59d7-49a4-a8e0-97888f2c387f\",\r\n \"subActivityId\": \"20c41293-ee06-43b0-9f85-1faf9b5ab846\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"0bf6b207-9c8c-43f0-a099-76d33cc8d11d\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;0d1d7012-9bec-42ca-94eb-41a4c8ac0207 ARID=7c88679c-59d7-49a4-a8e0-97888f2c387f \u003e GW.Http.CallContext/7c88679c-59d7-49a4-a8e0-97888f2c387f \u003e GW.ExecuteQuery/0bf6b207-9c8c-43f0-a099-76d33cc8d11d \u003e DN.FE.ExecuteQuery/20c41293-ee06-43b0-9f85-1faf9b5ab846)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:02:22.174]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:02:15.4920397Z);\nlet _endTime = datetime(2026-08-24T18:02:15.4920402Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hxrz24kgz22q'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:02:21.9924804Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;0d1d7012-9bec-42ca-94eb-41a4c8ac0207\",\r\n \"activityId\": \"7c88679c-59d7-49a4-a8e0-97888f2c387f\",\r\n \"subActivityId\": \"20c41293-ee06-43b0-9f85-1faf9b5ab846\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"0bf6b207-9c8c-43f0-a099-76d33cc8d11d\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;0d1d7012-9bec-42ca-94eb-41a4c8ac0207 ARID=7c88679c-59d7-49a4-a8e0-97888f2c387f \u003e GW.Http.CallContext/7c88679c-59d7-49a4-a8e0-97888f2c387f \u003e GW.ExecuteQuery/0bf6b207-9c8c-43f0-a099-76d33cc8d11d \u003e DN.FE.ExecuteQuery/20c41293-ee06-43b0-9f85-1faf9b5ab846)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[18:02:34.313]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:02:27.8203571Z);\nlet _endTime = datetime(2026-08-24T18:02:27.8203574Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hxrz24kgz22q'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:02:34.2861093Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;7b69ab3c-142e-42fc-ba7f-d284b2e71d38\",\r\n \"activityId\": \"833f4c61-46f8-4fdc-895d-d0121d895d64\",\r\n \"subActivityId\": \"18d5bb74-4c1c-4003-b6ff-d5c3cbb43607\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"cbd0d9b4-40b0-49a2-b3c0-ff7e4292d28e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;7b69ab3c-142e-42fc-ba7f-d284b2e71d38 ARID=833f4c61-46f8-4fdc-895d-d0121d895d64 \u003e GW.Http.CallContext/833f4c61-46f8-4fdc-895d-d0121d895d64 \u003e GW.ExecuteQuery/cbd0d9b4-40b0-49a2-b3c0-ff7e4292d28e \u003e DN.FE.ExecuteQuery/18d5bb74-4c1c-4003-b6ff-d5c3cbb43607)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[18:02:34.447]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T18:02:27.8203571Z);\nlet _endTime = datetime(2026-08-24T18:02:27.8203574Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-hxrz24kgz22q'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T18:02:34.2861093Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;7b69ab3c-142e-42fc-ba7f-d284b2e71d38\",\r\n \"activityId\": \"833f4c61-46f8-4fdc-895d-d0121d895d64\",\r\n \"subActivityId\": \"18d5bb74-4c1c-4003-b6ff-d5c3cbb43607\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"cbd0d9b4-40b0-49a2-b3c0-ff7e4292d28e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;7b69ab3c-142e-42fc-ba7f-d284b2e71d38 ARID=833f4c61-46f8-4fdc-895d-d0121d895d64 \u003e GW.Http.CallContext/833f4c61-46f8-4fdc-895d-d0121d895d64 \u003e GW.ExecuteQuery/cbd0d9b4-40b0-49a2-b3c0-ff7e4292d28e \u003e DN.FE.ExecuteQuery/18d5bb74-4c1c-4003-b6ff-d5c3cbb43607)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 16:40:39 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-hbs92nmx4z9w/cluster-candidate-5-0-v6w5lz should provision Unexpected error: <fmt.wrapError \| 0xc001a025e0>: failed to create HCP cluster cluster-candidate-5-0-v6w5lz: failed waiting for cluster="cluster-candidate-5-0-v6w5lz" in resourcegroup="rg-candidate-5-0-hbs92nmx4z9w" to finish creating: GET https://rp.j0629888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1af9d894-18ff-4381-9686-ac8da4e5e05a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1af9d894-18ff-4381-9686-ac8da4e5e05a", "name": "1af9d894-18ff-4381-9686-ac8da4e5e05a", "status": "Failed", "startTime": "2026-08-24T17:34:08.691049831Z", "endTime": "2026-08-24T17:53:13.772213433Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-hbs92nmx4z9w/cluster-candidate-5-0-v6w5lz should provision Unexpected error: <fmt.wrapError \| 0xc001a025e0>: failed to create HCP cluster cluster-candidate-5-0-v6w5lz: failed waiting for cluster="cluster-candidate-5-0-v6w5lz" in resourcegroup="rg-candidate-5-0-hbs92nmx4z9w" to finish creating: GET https://rp.j0629888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1af9d894-18ff-4381-9686-ac8da4e5e05a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1af9d894-18ff-4381-9686-ac8da4e5e05a", "name": "1af9d894-18ff-4381-9686-ac8da4e5e05a", "status": "Failed", "startTime": "2026-08-24T17:34:08.691049831Z", "endTime": "2026-08-24T17:53:13.772213433Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 16:40:39 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc0009626a0>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-2qrhl2np7wfb" to finish creating: GET https://rp.j0629888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ffc54622-63ff-4342-a7a6-0d895601d00c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ffc54622-63ff-4342-a7a6-0d895601d00c", "name": "ffc54622-63ff-4342-a7a6-0d895601d00c", "status": "Failed", "startTime": "2026-08-24T17:35:49.514492475Z", "endTime": "2026-08-24T17:54:53.594926153Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc0009626a0>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-2qrhl2np7wfb" to finish creating: GET https://rp.j0629888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ffc54622-63ff-4342-a7a6-0d895601d00c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ffc54622-63ff-4342-a7a6-0d895601d00c", "name": "ffc54622-63ff-4342-a7a6-0d895601d00c", "status": "Failed", "startTime": "2026-08-24T17:35:49.514492475Z", "endTime": "2026-08-24T17:54:53.594926153Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred cluster entered terminal Failed state after role assignment deployment 16:40:39 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.001s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.001s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... alert (2) alert [svc] FleetControllerRetryHotLoop fired 16:40:39 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FleetControllerRetryHotLoop does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-24T17:38:07Z Ended: 2026-08-24T17:45:05Z Severity: Sev3 Labels: alertname="FleetControllerRetryHotLoop", cluster="ci01-j0629888-svc", component="fleet", name="clustersserviceregistrationcontroller", severity="warning" Description: Fleet controller workqueue clustersserviceregistrationcontroller has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. alert [hcp] KubeStatefulSetReplicasMismatch fired 16:40:39 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-08-24T17:55:40Z Ended: 2026-08-24T18:14:40Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0629888-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.209:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0629888/providers/microsoft.monitor/accounts/hcps-j0629888", namespace="ocm-arohcpci01-2sd354mcvucng8mm6igmcq4k43caf9jb-s0d4l4d8t6j0p0t", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-tktc6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2sd354mcvucng8mm6igmcq4k43caf9jb-s0d4l4d8t6j0p0t/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 2: State: Resolved Started: 2026-08-24T17:55:40Z Ended: 2026-08-24T18:15:41Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0629888-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.209:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0629888/providers/microsoft.monitor/accounts/hcps-j0629888", namespace="ocm-arohcpci01-2sd354mcvucng8mm6igmcq4k43caf9jb-s0d4l4d8t6j0p0t", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-tktc6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2sd354mcvucng8mm6igmcq4k43caf9jb-s0d4l4d8t6j0p0t/etcd has not matched the expected number of replicas for longer than 15 minutes.
16:34:51	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 716e07d	alert, e2e	failed	★#6657 (open)	13	e2e (12) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 16:34:51 UTC · failed at e2e · test FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc001068580>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-ks59qpfw2b7j" to finish creating: GET https://rp.j2463616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/764346c6-fb81-4c96-9ce9-57d5acf0bfef -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/764346c6-fb81-4c96-9ce9-57d5acf0bfef", "name": "764346c6-fb81-4c96-9ce9-57d5acf0bfef", "status": "Failed", "startTime": "2026-08-24T17:28:32.803601268Z", "endTime": "2026-08-24T17:47:34.550152556Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc001068580>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-ks59qpfw2b7j" to finish creating: GET https://rp.j2463616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/764346c6-fb81-4c96-9ce9-57d5acf0bfef -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/764346c6-fb81-4c96-9ce9-57d5acf0bfef", "name": "764346c6-fb81-4c96-9ce9-57d5acf0bfef", "status": "Failed", "startTime": "2026-08-24T17:28:32.803601268Z", "endTime": "2026-08-24T17:47:34.550152556Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; provider Microsoft.RedHatOpenShift 16:34:51 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-cbhfzqtvb9th/cluster-candidate-4-21-5bbfxw should provision Unexpected error: <fmt.wrapError \| 0xc000ea2680>: failed to create HCP cluster cluster-candidate-4-21-5bbfxw: failed waiting for cluster="cluster-candidate-4-21-5bbfxw" in resourcegroup="rg-candidate-4-21-cbhfzqtvb9th" to finish creating: GET https://rp.j2463616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a86d82f7-93d6-43a5-99ee-59e2969fd318 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a86d82f7-93d6-43a5-99ee-59e2969fd318", "name": "a86d82f7-93d6-43a5-99ee-59e2969fd318", "status": "Failed", "startTime": "2026-08-24T17:28:32.844651935Z", "endTime": "2026-08-24T17:47:34.560561952Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-cbhfzqtvb9th/cluster-candidate-4-21-5bbfxw should provision Unexpected error: <fmt.wrapError \| 0xc000ea2680>: failed to create HCP cluster cluster-candidate-4-21-5bbfxw: failed waiting for cluster="cluster-candidate-4-21-5bbfxw" in resourcegroup="rg-candidate-4-21-cbhfzqtvb9th" to finish creating: GET https://rp.j2463616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a86d82f7-93d6-43a5-99ee-59e2969fd318 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a86d82f7-93d6-43a5-99ee-59e2969fd318", "name": "a86d82f7-93d6-43a5-99ee-59e2969fd318", "status": "Failed", "startTime": "2026-08-24T17:28:32.844651935Z", "endTime": "2026-08-24T17:47:34.560561952Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 16:34:51 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc000f64d80>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-5665zqlrt778" to finish creating: GET https://rp.j2463616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/049b9660-e1f7-4a39-9064-73a72d6b9292 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/049b9660-e1f7-4a39-9064-73a72d6b9292", "name": "049b9660-e1f7-4a39-9064-73a72d6b9292", "status": "Failed", "startTime": "2026-08-24T17:27:52.106192297Z", "endTime": "2026-08-24T17:46:54.569578114Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc000f64d80>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-5665zqlrt778" to finish creating: GET https://rp.j2463616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/049b9660-e1f7-4a39-9064-73a72d6b9292 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/049b9660-e1f7-4a39-9064-73a72d6b9292", "name": "049b9660-e1f7-4a39-9064-73a72d6b9292", "status": "Failed", "startTime": "2026-08-24T17:27:52.106192297Z", "endTime": "2026-08-24T17:46:54.569578114Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred stamp 2 scheduling must have requested resources 16:34:51 UTC · failed at e2e · test Fleet should have scheduling data for ready management clusters · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:162]: Timed out after 900.001s. scheduling data was not populated in time The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:155 with: stamp 2 scheduling must have requested resources Expected <v1.ResourceList \| len:0>: ... not to be empty fail [github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:162]: Timed out after 900.001s. scheduling data was not populated in time The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:155 with: stamp 2 scheduling must have requested resources Expected <v1.ResourceList \| len:0>: ... not to be empty ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 16:34:51 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-h6q45zq2k92c/cluster-candidate-4-22-ppq8j9 should provision Unexpected error: <fmt.wrapError \| 0xc001250700>: failed to create HCP cluster cluster-candidate-4-22-ppq8j9: failed waiting for cluster="cluster-candidate-4-22-ppq8j9" in resourcegroup="rg-candidate-4-22-h6q45zq2k92c" to finish creating: GET https://rp.j2463616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ceece4bb-0461-4918-a3ae-9721cd29423d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ceece4bb-0461-4918-a3ae-9721cd29423d", "name": "ceece4bb-0461-4918-a3ae-9721cd29423d", "status": "Failed", "startTime": "2026-08-24T17:28:32.643827373Z", "endTime": "2026-08-24T17:47:34.518536819Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-h6q45zq2k92c/cluster-candidate-4-22-ppq8j9 should provision Unexpected error: <fmt.wrapError \| 0xc001250700>: failed to create HCP cluster cluster-candidate-4-22-ppq8j9: failed waiting for cluster="cluster-candidate-4-22-ppq8j9" in resourcegroup="rg-candidate-4-22-h6q45zq2k92c" to finish creating: GET https://rp.j2463616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ceece4bb-0461-4918-a3ae-9721cd29423d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ceece4bb-0461-4918-a3ae-9721cd29423d", "name": "ceece4bb-0461-4918-a3ae-9721cd29423d", "status": "Failed", "startTime": "2026-08-24T17:28:32.643827373Z", "endTime": "2026-08-24T17:47:34.518536819Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 16:34:51 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--c84xq6vfx9vp/cluster-hs-pre-drcr6j should provision Unexpected error: <fmt.wrapError \| 0xc000dcc620>: failed to create HCP cluster cluster-hs-pre-drcr6j: failed waiting for cluster="cluster-hs-pre-drcr6j" in resourcegroup="rg-hs-pre--c84xq6vfx9vp" to finish creating: GET https://rp.j2463616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fb641809-c31d-45a1-b698-88b1574a5864 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fb641809-c31d-45a1-b698-88b1574a5864", "name": "fb641809-c31d-45a1-b698-88b1574a5864", "status": "Failed", "startTime": "2026-08-24T17:28:32.693254211Z", "endTime": "2026-08-24T17:47:34.559332583Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--c84xq6vfx9vp/cluster-hs-pre-drcr6j should provision Unexpected error: <fmt.wrapError \| 0xc000dcc620>: failed to create HCP cluster cluster-hs-pre-drcr6j: failed waiting for cluster="cluster-hs-pre-drcr6j" in resourcegroup="rg-hs-pre--c84xq6vfx9vp" to finish creating: GET https://rp.j2463616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fb641809-c31d-45a1-b698-88b1574a5864 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fb641809-c31d-45a1-b698-88b1574a5864", "name": "fb641809-c31d-45a1-b698-88b1574a5864", "status": "Failed", "startTime": "2026-08-24T17:28:32.693254211Z", "endTime": "2026-08-24T17:47:34.559332583Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 16:34:51 UTC · failed at e2e · test SRE can pause schedules to stop backup execution for an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc0017fe160>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-w9jj96cfm4kd" to finish creating: GET https://rp.j2463616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/925e4a5c-4a74-40da-88a8-0093ce630809 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/925e4a5c-4a74-40da-88a8-0093ce630809", "name": "925e4a5c-4a74-40da-88a8-0093ce630809", "status": "Failed", "startTime": "2026-08-24T17:28:33.595393096Z", "endTime": "2026-08-24T17:47:34.559744671Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc0017fe160>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-w9jj96cfm4kd" to finish creating: GET https://rp.j2463616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/925e4a5c-4a74-40da-88a8-0093ce630809 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/925e4a5c-4a74-40da-88a8-0093ce630809", "name": "925e4a5c-4a74-40da-88a8-0093ce630809", "status": "Failed", "startTime": "2026-08-24T17:28:33.595393096Z", "endTime": "2026-08-24T17:47:34.559744671Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 16:34:51 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128" Unexpected error: <fmt.wrapError \| 0xc00123e240>: failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-2xcnpkr48kxk" to finish creating: GET https://rp.j2463616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3185d8c9-e246-4082-b170-898da28192d1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3185d8c9-e246-4082-b170-898da28192d1", "name": "3185d8c9-e246-4082-b170-898da28192d1", "status": "Failed", "startTime": "2026-08-24T17:28:32.888311428Z", "endTime": "2026-08-24T17:47:34.534131157Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128" Unexpected error: <fmt.wrapError \| 0xc00123e240>: failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-2xcnpkr48kxk" to finish creating: GET https://rp.j2463616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3185d8c9-e246-4082-b170-898da28192d1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3185d8c9-e246-4082-b170-898da28192d1", "name": "3185d8c9-e246-4082-b170-898da28192d1", "status": "Failed", "startTime": "2026-08-24T17:28:32.888311428Z", "endTime": "2026-08-24T17:47:34.534131157Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 16:34:51 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc0010e6ae0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-85shlfj9dj9q" to finish creating: GET https://rp.j2463616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f6727d04-f4f1-41ce-9763-ca7ee1e46ac5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f6727d04-f4f1-41ce-9763-ca7ee1e46ac5", "name": "f6727d04-f4f1-41ce-9763-ca7ee1e46ac5", "status": "Failed", "startTime": "2026-08-24T17:29:34.226726362Z", "endTime": "2026-08-24T17:48:34.53600893Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc0010e6ae0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-85shlfj9dj9q" to finish creating: GET https://rp.j2463616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f6727d04-f4f1-41ce-9763-ca7ee1e46ac5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f6727d04-f4f1-41ce-9763-ca7ee1e46ac5", "name": "f6727d04-f4f1-41ce-9763-ca7ee1e46ac5", "status": "Failed", "startTime": "2026-08-24T17:29:34.226726362Z", "endTime": "2026-08-24T17:48:34.53600893Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred query-basic: command failed: exit status 1 16:34:51 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.009s. Expected success, but got an error: <errors.errorString \| 0xc001b88230>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[17:51:07.885]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T17:51:04.0328787Z);\nlet _endTime = datetime(2026-08-24T17:51:04.0328790Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6r7jxj55gs7j'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T17:51:07.8639099Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;6e885cb8-5f9f-4176-bf4b-a6104ddaeca6\",\r\n \"activityId\": \"2b6dc4b8-71a6-45c2-8c84-a98044fb6c9c\",\r\n \"subActivityId\": \"dc466457-5534-436d-b9ae-dc5fd54f794d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"77ac7d84-b914-471e-9b71-1a82ec5cc8a3\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;6e885cb8-5f9f-4176-bf4b-a6104ddaeca6 ARID=2b6dc4b8-71a6-45c2-8c84-a98044fb6c9c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f011e5f0-7470-4963-a09c-e21fd80fe1b0 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f011e5f0-7470-4963-a09c-e21fd80fe1b0 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/77ac7d84-b914-471e-9b71-1a82ec5cc8a3 \u003e DN.FE.ExecuteQuery/dc466457-5534-436d-b9ae-dc5fd54f794d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[17:51:07.942]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T17:51:04.0328787Z);\nlet _endTime = datetime(2026-08-24T17:51:04.0328790Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6r7jxj55gs7j'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T17:51:07.8639099Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;6e885cb8-5f9f-4176-bf4b-a6104ddaeca6\",\r\n \"activityId\": \"2b6dc4b8-71a6-45c2-8c84-a98044fb6c9c\",\r\n \"subActivityId\": \"dc466457-5534-436d-b9ae-dc5fd54f794d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"77ac7d84-b914-471e-9b71-1a82ec5cc8a3\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;6e885cb8-5f9f-4176-bf4b-a6104ddaeca6 ARID=2b6dc4b8-71a6-45c2-8c84-a98044fb6c9c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f011e5f0-7470-4963-a09c-e21fd80fe1b0 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f011e5f0-7470-4963-a09c-e21fd80fe1b0 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/77ac7d84-b914-471e-9b71-1a82ec5cc8a3 \u003e DN.FE.ExecuteQuery/dc466457-5534-436d-b9ae-dc5fd54f794d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[17:51:11.812]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T17:51:08.0060747Z);\nlet _endTime = datetime(2026-08-24T17:51:08.0060750Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6r7jxj55gs7j'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T17:51:11.7874017Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;e5b94b99-89fd-40ca-83cf-ccc36b30573b\",\r\n \"activityId\": \"580e6639-eed5-43ed-b0d5-a73c66a2b85e\",\r\n \"subActivityId\": \"4b5d648b-cb9f-4310-a66c-e93ce0e38547\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"2e63fd90-d856-490b-b345-ef00054c44aa\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;e5b94b99-89fd-40ca-83cf-ccc36b30573b ARID=580e6639-eed5-43ed-b0d5-a73c66a2b85e \u003e GW.Http.CallContext/580e6639-eed5-43ed-b0d5-a73c66a2b85e \u003e GW.ExecuteQuery/2e63fd90-d856-490b-b345-ef00054c44aa \u003e DN.FE.ExecuteQuery/4b5d648b-cb9f-4310-a66c-e93ce0e38547)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[17:51:11.896]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T17:51:08.0060747Z);\nlet _endTime = datetime(2026-08-24T17:51:08.0060750Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6r7jxj55gs7j'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T17:51:11.7874017Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;e5b94b99-89fd-40ca-83cf-ccc36b30573b\",\r\n \"activityId\": \"580e6639-eed5-43ed-b0d5-a73c66a2b85e\",\r\n \"subActivityId\": \"4b5d648b-cb9f-4310-a66c-e93ce0e38547\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"2e63fd90-d856-490b-b345-ef00054c44aa\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;e5b94b99-89fd-40ca-83cf-ccc36b30573b ARID=580e6639-eed5-43ed-b0d5-a73c66a2b85e \u003e GW.Http.CallContext/580e6639-eed5-43ed-b0d5-a73c66a2b85e \u003e GW.ExecuteQuery/2e63fd90-d856-490b-b345-ef00054c44aa \u003e DN.FE.ExecuteQuery/4b5d648b-cb9f-4310-a66c-e93ce0e38547)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[17:51:19.503]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T17:51:15.9721690Z);\nlet _endTime = datetime(2026-08-24T17:51:15.9721694Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6r7jxj55gs7j'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T17:51:19.4777940Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;c86aa40d-117d-4a61-a0bd-780e2aa3a1d5\",\r\n \"activityId\": \"a72d1187-0185-4d6b-b98b-6308c9ee1cbc\",\r\n \"subActivityId\": \"9c08dcb8-ca10-470a-b3dc-81e2eb738b18\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"b23cbcab-d834-4c81-85fe-cbbda2dc2556\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;c86aa40d-117d-4a61-a0bd-780e2aa3a1d5 ARID=a72d1187-0185-4d6b-b98b-6308c9ee1cbc \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/9083dda2-6e55-416d-bec2-c70acc01aed8 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/9083dda2-6e55-416d-bec2-c70acc01aed8 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/b23cbcab-d834-4c81-85fe-cbbda2dc2556 \u003e DN.FE.ExecuteQuery/9c08dcb8-ca10-470a-b3dc-81e2eb738b18)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[17:51:19.625]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T17:51:15.9721690Z);\nlet _endTime = datetime(2026-08-24T17:51:15.9721694Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6r7jxj55gs7j'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T17:51:19.4777940Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;c86aa40d-117d-4a61-a0bd-780e2aa3a1d5\",\r\n \"activityId\": \"a72d1187-0185-4d6b-b98b-6308c9ee1cbc\",\r\n \"subActivityId\": \"9c08dcb8-ca10-470a-b3dc-81e2eb738b18\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"b23cbcab-d834-4c81-85fe-cbbda2dc2556\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;c86aa40d-117d-4a61-a0bd-780e2aa3a1d5 ARID=a72d1187-0185-4d6b-b98b-6308c9ee1cbc \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/9083dda2-6e55-416d-bec2-c70acc01aed8 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/9083dda2-6e55-416d-bec2-c70acc01aed8 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/b23cbcab-d834-4c81-85fe-cbbda2dc2556 \u003e DN.FE.ExecuteQuery/9c08dcb8-ca10-470a-b3dc-81e2eb738b18)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.009s. Expected success, but got an error: <errors.errorString \| 0xc001b88230>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[17:51:07.885]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T17:51:04.0328787Z);\nlet _endTime = datetime(2026-08-24T17:51:04.0328790Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6r7jxj55gs7j'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T17:51:07.8639099Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;6e885cb8-5f9f-4176-bf4b-a6104ddaeca6\",\r\n \"activityId\": \"2b6dc4b8-71a6-45c2-8c84-a98044fb6c9c\",\r\n \"subActivityId\": \"dc466457-5534-436d-b9ae-dc5fd54f794d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"77ac7d84-b914-471e-9b71-1a82ec5cc8a3\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;6e885cb8-5f9f-4176-bf4b-a6104ddaeca6 ARID=2b6dc4b8-71a6-45c2-8c84-a98044fb6c9c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f011e5f0-7470-4963-a09c-e21fd80fe1b0 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f011e5f0-7470-4963-a09c-e21fd80fe1b0 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/77ac7d84-b914-471e-9b71-1a82ec5cc8a3 \u003e DN.FE.ExecuteQuery/dc466457-5534-436d-b9ae-dc5fd54f794d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[17:51:07.942]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T17:51:04.0328787Z);\nlet _endTime = datetime(2026-08-24T17:51:04.0328790Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6r7jxj55gs7j'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T17:51:07.8639099Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;6e885cb8-5f9f-4176-bf4b-a6104ddaeca6\",\r\n \"activityId\": \"2b6dc4b8-71a6-45c2-8c84-a98044fb6c9c\",\r\n \"subActivityId\": \"dc466457-5534-436d-b9ae-dc5fd54f794d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"77ac7d84-b914-471e-9b71-1a82ec5cc8a3\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;6e885cb8-5f9f-4176-bf4b-a6104ddaeca6 ARID=2b6dc4b8-71a6-45c2-8c84-a98044fb6c9c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f011e5f0-7470-4963-a09c-e21fd80fe1b0 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f011e5f0-7470-4963-a09c-e21fd80fe1b0 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/77ac7d84-b914-471e-9b71-1a82ec5cc8a3 \u003e DN.FE.ExecuteQuery/dc466457-5534-436d-b9ae-dc5fd54f794d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[17:51:11.812]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T17:51:08.0060747Z);\nlet _endTime = datetime(2026-08-24T17:51:08.0060750Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6r7jxj55gs7j'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T17:51:11.7874017Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;e5b94b99-89fd-40ca-83cf-ccc36b30573b\",\r\n \"activityId\": \"580e6639-eed5-43ed-b0d5-a73c66a2b85e\",\r\n \"subActivityId\": \"4b5d648b-cb9f-4310-a66c-e93ce0e38547\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"2e63fd90-d856-490b-b345-ef00054c44aa\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;e5b94b99-89fd-40ca-83cf-ccc36b30573b ARID=580e6639-eed5-43ed-b0d5-a73c66a2b85e \u003e GW.Http.CallContext/580e6639-eed5-43ed-b0d5-a73c66a2b85e \u003e GW.ExecuteQuery/2e63fd90-d856-490b-b345-ef00054c44aa \u003e DN.FE.ExecuteQuery/4b5d648b-cb9f-4310-a66c-e93ce0e38547)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[17:51:11.896]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T17:51:08.0060747Z);\nlet _endTime = datetime(2026-08-24T17:51:08.0060750Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6r7jxj55gs7j'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T17:51:11.7874017Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;e5b94b99-89fd-40ca-83cf-ccc36b30573b\",\r\n \"activityId\": \"580e6639-eed5-43ed-b0d5-a73c66a2b85e\",\r\n \"subActivityId\": \"4b5d648b-cb9f-4310-a66c-e93ce0e38547\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"2e63fd90-d856-490b-b345-ef00054c44aa\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;e5b94b99-89fd-40ca-83cf-ccc36b30573b ARID=580e6639-eed5-43ed-b0d5-a73c66a2b85e \u003e GW.Http.CallContext/580e6639-eed5-43ed-b0d5-a73c66a2b85e \u003e GW.ExecuteQuery/2e63fd90-d856-490b-b345-ef00054c44aa \u003e DN.FE.ExecuteQuery/4b5d648b-cb9f-4310-a66c-e93ce0e38547)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[17:51:19.503]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T17:51:15.9721690Z);\nlet _endTime = datetime(2026-08-24T17:51:15.9721694Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6r7jxj55gs7j'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T17:51:19.4777940Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;c86aa40d-117d-4a61-a0bd-780e2aa3a1d5\",\r\n \"activityId\": \"a72d1187-0185-4d6b-b98b-6308c9ee1cbc\",\r\n \"subActivityId\": \"9c08dcb8-ca10-470a-b3dc-81e2eb738b18\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"b23cbcab-d834-4c81-85fe-cbbda2dc2556\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;c86aa40d-117d-4a61-a0bd-780e2aa3a1d5 ARID=a72d1187-0185-4d6b-b98b-6308c9ee1cbc \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/9083dda2-6e55-416d-bec2-c70acc01aed8 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/9083dda2-6e55-416d-bec2-c70acc01aed8 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/b23cbcab-d834-4c81-85fe-cbbda2dc2556 \u003e DN.FE.ExecuteQuery/9c08dcb8-ca10-470a-b3dc-81e2eb738b18)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[17:51:19.625]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T17:51:15.9721690Z);\nlet _endTime = datetime(2026-08-24T17:51:15.9721694Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6r7jxj55gs7j'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T17:51:19.4777940Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;c86aa40d-117d-4a61-a0bd-780e2aa3a1d5\",\r\n \"activityId\": \"a72d1187-0185-4d6b-b98b-6308c9ee1cbc\",\r\n \"subActivityId\": \"9c08dcb8-ca10-470a-b3dc-81e2eb738b18\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"b23cbcab-d834-4c81-85fe-cbbda2dc2556\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;c86aa40d-117d-4a61-a0bd-780e2aa3a1d5 ARID=a72d1187-0185-4d6b-b98b-6308c9ee1cbc \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/9083dda2-6e55-416d-bec2-c70acc01aed8 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/9083dda2-6e55-416d-bec2-c70acc01aed8 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/b23cbcab-d834-4c81-85fe-cbbda2dc2556 \u003e DN.FE.ExecuteQuery/9c08dcb8-ca10-470a-b3dc-81e2eb738b18)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... cluster entered terminal Failed state after role assignment deployment 16:34:51 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.002s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.002s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 16:34:51 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-ft86lhtlb8k7/cluster-candidate-4-20-zsqmmf should provision Unexpected error: <fmt.wrapError \| 0xc0013e6360>: failed to create HCP cluster cluster-candidate-4-20-zsqmmf: failed waiting for cluster="cluster-candidate-4-20-zsqmmf" in resourcegroup="rg-candidate-4-20-ft86lhtlb8k7" to finish creating: GET https://rp.j2463616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/52586d05-b59b-44b4-8d37-e62438c994b8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/52586d05-b59b-44b4-8d37-e62438c994b8", "name": "52586d05-b59b-44b4-8d37-e62438c994b8", "status": "Failed", "startTime": "2026-08-24T17:28:33.090044167Z", "endTime": "2026-08-24T17:47:34.534082805Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-ft86lhtlb8k7/cluster-candidate-4-20-zsqmmf should provision Unexpected error: <fmt.wrapError \| 0xc0013e6360>: failed to create HCP cluster cluster-candidate-4-20-zsqmmf: failed waiting for cluster="cluster-candidate-4-20-zsqmmf" in resourcegroup="rg-candidate-4-20-ft86lhtlb8k7" to finish creating: GET https://rp.j2463616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/52586d05-b59b-44b4-8d37-e62438c994b8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/52586d05-b59b-44b4-8d37-e62438c994b8", "name": "52586d05-b59b-44b4-8d37-e62438c994b8", "status": "Failed", "startTime": "2026-08-24T17:28:33.090044167Z", "endTime": "2026-08-24T17:47:34.534082805Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 16:34:51 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 21 time(s) (1 unknown, 20 known) Firing 1: State: Resolved Started: 2026-08-24T17:41:05Z Ended: 2026-08-24T18:03:06Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2463616-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd32h2d1lquds92aqdmsnijjli4i6pt-f4l8d4z1z4r3u1t", pod="router-7685ff5cfb-4r4wg", severity="warning" Description: Pod ocm-arohcpci01-2sd32h2d1lquds92aqdmsnijjli4i6pt-f4l8d4z1z4r3u1t/router-7685ff5cfb-4r4wg has been in a non-ready state for longer than 5 minutes.
16:32:12	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 9749b2d	provision	failed	#4566 (open)	1	provision (1) ERROR CODE: DeploymentNotFound 16:32:12 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Fleet.Registration/management/output (stamp=1) · suite step graph Show raw failure time=2026-08-24T17:20:28.034Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=management step=output stamp=1 description="Step output\n Kind: ARM\n Template: ../../dev-infrastructure/templates/output-mgmt.bicep\n Parameters: ../../dev-infrastructure/configurations/output-mgmt.tmpl.bicepparam" time=2026-08-24T17:20:28.426Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=management step=output stamp=1 time=2026-08-24T17:20:28.817Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=management step=output stamp=1 deployment=3472b226dd1bf3f5c2c2fbd8f62937b0fe2f97cb4ca8a63168999f9ba0945d23 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2076032-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F3472b226dd1bf3f5c2c2fbd8f62937b0fe2f97cb4ca8a63168999f9ba0945d23 time=2026-08-24T17:20:28.866Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=management step=output stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2076032-mgmt-1/providers/Microsoft.Resources/deployments/3472b226dd1bf3f5c2c2fbd8f62937b0fe2f97cb4ca8a63168999f9ba0945d23/operationStatuses/08584140148567905734\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DeploymentNotFound\",\n \"message\": \"Deployment '3472b226dd1bf3f5c2c2fbd8f62937b0fe2f97cb4ca8a63168999f9ba0945d23' could not be found.\"\n }\n}\n--------------------------------------------------------------------------------\n"
16:21:04	pull-ci-Azure-ARO-HCP-main-e2e-parallel head ccc0138	e2e	failed	★#6639 (open)	1	e2e (1) query-basic: command failed: exit status 1 16:21:04 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.001s. Expected success, but got an error: <errors.errorString \| 0xc0009863f0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[17:50:10.830]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T17:50:07.0974750Z);\nlet _endTime = datetime(2026-08-24T17:50:07.0974754Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bsdxktxx588n'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T17:50:10.8083087Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;fac29151-0f82-45b0-b39e-1b80a12094a4\",\r\n \"activityId\": \"578602da-c7bd-4612-b601-5ddeff2f3885\",\r\n \"subActivityId\": \"4dee2970-8e9b-48d4-bf21-5bfe620f649d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"d1fdc108-83eb-4130-b339-83dd3ff9fcbb\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;fac29151-0f82-45b0-b39e-1b80a12094a4 ARID=578602da-c7bd-4612-b601-5ddeff2f3885 \u003e GW.Http.CallContext/578602da-c7bd-4612-b601-5ddeff2f3885 \u003e GW.ExecuteQuery/d1fdc108-83eb-4130-b339-83dd3ff9fcbb \u003e DN.FE.ExecuteQuery/4dee2970-8e9b-48d4-bf21-5bfe620f649d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[17:50:10.904]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T17:50:07.0974750Z);\nlet _endTime = datetime(2026-08-24T17:50:07.0974754Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bsdxktxx588n'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T17:50:10.8083087Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;fac29151-0f82-45b0-b39e-1b80a12094a4\",\r\n \"activityId\": \"578602da-c7bd-4612-b601-5ddeff2f3885\",\r\n \"subActivityId\": \"4dee2970-8e9b-48d4-bf21-5bfe620f649d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"d1fdc108-83eb-4130-b339-83dd3ff9fcbb\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;fac29151-0f82-45b0-b39e-1b80a12094a4 ARID=578602da-c7bd-4612-b601-5ddeff2f3885 \u003e GW.Http.CallContext/578602da-c7bd-4612-b601-5ddeff2f3885 \u003e GW.ExecuteQuery/d1fdc108-83eb-4130-b339-83dd3ff9fcbb \u003e DN.FE.ExecuteQuery/4dee2970-8e9b-48d4-bf21-5bfe620f649d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[17:50:14.600]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T17:50:10.9799365Z);\nlet _endTime = datetime(2026-08-24T17:50:10.9799369Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bsdxktxx588n'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T17:50:14.5738009Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;9cfffa2e-fd47-4de8-b850-fcb128ae3957\",\r\n \"activityId\": \"b1a9e4f8-fe9f-43eb-98c4-295226684f99\",\r\n \"subActivityId\": \"5346eafa-957d-49ee-b78d-d8e0622b2808\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"ea635fed-0287-4459-9508-ca4085e8c3aa\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;9cfffa2e-fd47-4de8-b850-fcb128ae3957 ARID=b1a9e4f8-fe9f-43eb-98c4-295226684f99 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/33092b63-29bb-42fc-b88a-af6bae1ef15f \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/33092b63-29bb-42fc-b88a-af6bae1ef15f \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/ea635fed-0287-4459-9508-ca4085e8c3aa \u003e DN.FE.ExecuteQuery/5346eafa-957d-49ee-b78d-d8e0622b2808)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[17:50:14.688]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T17:50:10.9799365Z);\nlet _endTime = datetime(2026-08-24T17:50:10.9799369Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bsdxktxx588n'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T17:50:14.5738009Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;9cfffa2e-fd47-4de8-b850-fcb128ae3957\",\r\n \"activityId\": \"b1a9e4f8-fe9f-43eb-98c4-295226684f99\",\r\n \"subActivityId\": \"5346eafa-957d-49ee-b78d-d8e0622b2808\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"ea635fed-0287-4459-9508-ca4085e8c3aa\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;9cfffa2e-fd47-4de8-b850-fcb128ae3957 ARID=b1a9e4f8-fe9f-43eb-98c4-295226684f99 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/33092b63-29bb-42fc-b88a-af6bae1ef15f \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/33092b63-29bb-42fc-b88a-af6bae1ef15f \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/ea635fed-0287-4459-9508-ca4085e8c3aa \u003e DN.FE.ExecuteQuery/5346eafa-957d-49ee-b78d-d8e0622b2808)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[17:50:21.769]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T17:50:18.2612887Z);\nlet _endTime = datetime(2026-08-24T17:50:18.2612891Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bsdxktxx588n'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T17:50:21.7433795Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;8abde100-7c98-4f34-a0a5-8f66c337da8e\",\r\n \"activityId\": \"a71d2be1-e393-45cd-a6e4-5527dafdf569\",\r\n \"subActivityId\": \"71101ab2-ff83-4119-8e2b-ee21aaceb2d6\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"dcbe4a01-596d-4348-affe-223b58329c91\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;8abde100-7c98-4f34-a0a5-8f66c337da8e ARID=a71d2be1-e393-45cd-a6e4-5527dafdf569 \u003e GW.Http.CallContext/a71d2be1-e393-45cd-a6e4-5527dafdf569 \u003e GW.ExecuteQuery/dcbe4a01-596d-4348-affe-223b58329c91 \u003e DN.FE.ExecuteQuery/71101ab2-ff83-4119-8e2b-ee21aaceb2d6)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[17:50:21.932]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T17:50:18.2612887Z);\nlet _endTime = datetime(2026-08-24T17:50:18.2612891Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bsdxktxx588n'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T17:50:21.7433795Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;8abde100-7c98-4f34-a0a5-8f66c337da8e\",\r\n \"activityId\": \"a71d2be1-e393-45cd-a6e4-5527dafdf569\",\r\n \"subActivityId\": \"71101ab2-ff83-4119-8e2b-ee21aaceb2d6\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"dcbe4a01-596d-4348-affe-223b58329c91\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;8abde100-7c98-4f34-a0a5-8f66c337da8e ARID=a71d2be1-e393-45cd-a6e4-5527dafdf569 \u003e GW.Http.CallContext/a71d2be1-e393-45cd-a6e4-5527dafdf569 \u003e GW.ExecuteQuery/dcbe4a01-596d-4348-affe-223b58329c91 \u003e DN.FE.ExecuteQuery/71101ab2-ff83-4119-8e2b-ee21aaceb2d6)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.001s. Expected success, but got an error: <errors.errorString \| 0xc0009863f0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[17:50:10.830]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T17:50:07.0974750Z);\nlet _endTime = datetime(2026-08-24T17:50:07.0974754Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bsdxktxx588n'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T17:50:10.8083087Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;fac29151-0f82-45b0-b39e-1b80a12094a4\",\r\n \"activityId\": \"578602da-c7bd-4612-b601-5ddeff2f3885\",\r\n \"subActivityId\": \"4dee2970-8e9b-48d4-bf21-5bfe620f649d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"d1fdc108-83eb-4130-b339-83dd3ff9fcbb\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;fac29151-0f82-45b0-b39e-1b80a12094a4 ARID=578602da-c7bd-4612-b601-5ddeff2f3885 \u003e GW.Http.CallContext/578602da-c7bd-4612-b601-5ddeff2f3885 \u003e GW.ExecuteQuery/d1fdc108-83eb-4130-b339-83dd3ff9fcbb \u003e DN.FE.ExecuteQuery/4dee2970-8e9b-48d4-bf21-5bfe620f649d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[17:50:10.904]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T17:50:07.0974750Z);\nlet _endTime = datetime(2026-08-24T17:50:07.0974754Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bsdxktxx588n'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T17:50:10.8083087Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;fac29151-0f82-45b0-b39e-1b80a12094a4\",\r\n \"activityId\": \"578602da-c7bd-4612-b601-5ddeff2f3885\",\r\n \"subActivityId\": \"4dee2970-8e9b-48d4-bf21-5bfe620f649d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"d1fdc108-83eb-4130-b339-83dd3ff9fcbb\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;fac29151-0f82-45b0-b39e-1b80a12094a4 ARID=578602da-c7bd-4612-b601-5ddeff2f3885 \u003e GW.Http.CallContext/578602da-c7bd-4612-b601-5ddeff2f3885 \u003e GW.ExecuteQuery/d1fdc108-83eb-4130-b339-83dd3ff9fcbb \u003e DN.FE.ExecuteQuery/4dee2970-8e9b-48d4-bf21-5bfe620f649d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[17:50:14.600]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T17:50:10.9799365Z);\nlet _endTime = datetime(2026-08-24T17:50:10.9799369Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bsdxktxx588n'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T17:50:14.5738009Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;9cfffa2e-fd47-4de8-b850-fcb128ae3957\",\r\n \"activityId\": \"b1a9e4f8-fe9f-43eb-98c4-295226684f99\",\r\n \"subActivityId\": \"5346eafa-957d-49ee-b78d-d8e0622b2808\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"ea635fed-0287-4459-9508-ca4085e8c3aa\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;9cfffa2e-fd47-4de8-b850-fcb128ae3957 ARID=b1a9e4f8-fe9f-43eb-98c4-295226684f99 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/33092b63-29bb-42fc-b88a-af6bae1ef15f \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/33092b63-29bb-42fc-b88a-af6bae1ef15f \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/ea635fed-0287-4459-9508-ca4085e8c3aa \u003e DN.FE.ExecuteQuery/5346eafa-957d-49ee-b78d-d8e0622b2808)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[17:50:14.688]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T17:50:10.9799365Z);\nlet _endTime = datetime(2026-08-24T17:50:10.9799369Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bsdxktxx588n'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T17:50:14.5738009Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;9cfffa2e-fd47-4de8-b850-fcb128ae3957\",\r\n \"activityId\": \"b1a9e4f8-fe9f-43eb-98c4-295226684f99\",\r\n \"subActivityId\": \"5346eafa-957d-49ee-b78d-d8e0622b2808\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"ea635fed-0287-4459-9508-ca4085e8c3aa\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;9cfffa2e-fd47-4de8-b850-fcb128ae3957 ARID=b1a9e4f8-fe9f-43eb-98c4-295226684f99 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/33092b63-29bb-42fc-b88a-af6bae1ef15f \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/33092b63-29bb-42fc-b88a-af6bae1ef15f \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/ea635fed-0287-4459-9508-ca4085e8c3aa \u003e DN.FE.ExecuteQuery/5346eafa-957d-49ee-b78d-d8e0622b2808)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[17:50:21.769]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T17:50:18.2612887Z);\nlet _endTime = datetime(2026-08-24T17:50:18.2612891Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bsdxktxx588n'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T17:50:21.7433795Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;8abde100-7c98-4f34-a0a5-8f66c337da8e\",\r\n \"activityId\": \"a71d2be1-e393-45cd-a6e4-5527dafdf569\",\r\n \"subActivityId\": \"71101ab2-ff83-4119-8e2b-ee21aaceb2d6\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"dcbe4a01-596d-4348-affe-223b58329c91\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;8abde100-7c98-4f34-a0a5-8f66c337da8e ARID=a71d2be1-e393-45cd-a6e4-5527dafdf569 \u003e GW.Http.CallContext/a71d2be1-e393-45cd-a6e4-5527dafdf569 \u003e GW.ExecuteQuery/dcbe4a01-596d-4348-affe-223b58329c91 \u003e DN.FE.ExecuteQuery/71101ab2-ff83-4119-8e2b-ee21aaceb2d6)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[17:50:21.932]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-23T17:50:18.2612887Z);\nlet _endTime = datetime(2026-08-24T17:50:18.2612891Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bsdxktxx588n'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-24T17:50:21.7433795Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;8abde100-7c98-4f34-a0a5-8f66c337da8e\",\r\n \"activityId\": \"a71d2be1-e393-45cd-a6e4-5527dafdf569\",\r\n \"subActivityId\": \"71101ab2-ff83-4119-8e2b-ee21aaceb2d6\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"dcbe4a01-596d-4348-affe-223b58329c91\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;8abde100-7c98-4f34-a0a5-8f66c337da8e ARID=a71d2be1-e393-45cd-a6e4-5527dafdf569 \u003e GW.Http.CallContext/a71d2be1-e393-45cd-a6e4-5527dafdf569 \u003e GW.ExecuteQuery/dcbe4a01-596d-4348-affe-223b58329c91 \u003e DN.FE.ExecuteQuery/71101ab2-ff83-4119-8e2b-ee21aaceb2d6)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ...
16:10:41	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c403afe	alert	failed	#6638 (open)	1	alert (1) alert [svc] FrontendPathMedianLatency fired 16:10:41 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-24T17:21:19Z Ended: 2026-08-24T17:40:17Z Severity: Sev3 Labels: alertname="FrontendPathMedianLatency", cluster="ci01-j9342336-svc", component="frontend", method="put", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning" Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j9342336-svc).
16:06:36	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c9b8452	alert, e2e	failed	⚠#6584 (open)	2	e2e (1) ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator]; provider Microsoft.RedHatOpenShift 16:06:36 UTC · failed at e2e · test SRE can pause schedules to stop backup execution for an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000ef90e0>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-2zg88lxq9q4w" to finish creating: GET https://rp.j3558656.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b976f9f3-2891-4433-868a-76a8d23de317 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b976f9f3-2891-4433-868a-76a8d23de317", "name": "b976f9f3-2891-4433-868a-76a8d23de317", "status": "Failed", "startTime": "2026-08-24T16:54:08.369328068Z", "endTime": "2026-08-24T17:13:08.208412685Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000ef90e0>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-2zg88lxq9q4w" to finish creating: GET https://rp.j3558656.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b976f9f3-2891-4433-868a-76a8d23de317 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b976f9f3-2891-4433-868a-76a8d23de317", "name": "b976f9f3-2891-4433-868a-76a8d23de317", "status": "Failed", "startTime": "2026-08-24T16:54:08.369328068Z", "endTime": "2026-08-24T17:13:08.208412685Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 16:06:36 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-24T17:05:25Z Ended: 2026-08-24T17:12:23Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3558656-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd2i4t2jotejuk2memhr38s3p4vth8f-n6e5i1i6m0d7n8a", pod="router-6597db7-qp7zx", severity="warning" Description: Pod ocm-arohcpci01-2sd2i4t2jotejuk2memhr38s3p4vth8f-n6e5i1i6m0d7n8a/router-6597db7-qp7zx has been in a non-ready state for longer than 5 minutes.
15:56:46	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 3c81c74	n/a	passed	#6645 (open)	0	n/a
15:56:41	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 515e34b	n/a	passed	#6583 (open)	0	n/a
15:52:12	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 33f9055 · merge 33f9055	n/a	passed	#6278 (merged)	0	n/a
15:47:54	pull-ci-Azure-ARO-HCP-main-e2e-parallel head af0ab1f	n/a	passed	#6504 (open)	0	n/a
15:43:31	pull-ci-Azure-ARO-HCP-main-e2e-parallel head bdc8996	provision	failed	⚠#6659 (open)	1	provision (1) ERROR CODE: DeploymentFailed; detail code StorageAccountOperationInProgress; detail message An operation is currently performing on this storage account that requires exclusive access. 15:43:31 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/infra (stamp=2) · suite step graph Show raw failure time=2026-08-24T16:09:49.176Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=2 description="Step infra\n Kind: ARM\n Template: templates/mgmt-infra.bicep\n Parameters: configurations/mgmt-infra.tmpl.bicepparam" time=2026-08-24T16:09:49.840Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=2 time=2026-08-24T16:09:50.553Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=2 deployment=c783e3e3d697ee4d22dffbab4271479e105f154955a91c3a47e9819df57ac290 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1968512-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fc783e3e3d697ee4d22dffbab4271479e105f154955a91c3a47e9819df57ac290 time=2026-08-24T16:11:20.899Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1968512-mgmt-2/providers/Microsoft.Resources/deployments/c783e3e3d697ee4d22dffbab4271479e105f154955a91c3a47e9819df57ac290/operationStatuses/08584140190952500816\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1968512-mgmt-2/providers/Microsoft.Resources/deployments/hcp-backups-storage\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1968512-mgmt-2/providers/Microsoft.Resources/deployments/hcpBackupsStorageAccount\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1968512-mgmt-2/providers/Microsoft.Resources/deployments/hcpBackupsStorageAccount\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"StorageAccountOperationInProgress\\\",\\r\\n \\\"message\\\": \\\"An operation is currently performing on this storage account that requires exclusive access.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
15:33:54	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 00c069f	n/a	passed	#6363 (open)	0	n/a
15:09:12	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 2f68bd3	unknown	failed	#6070 (open)	1	non-artifact-backed failure (no junit artifacts)
14:50:52	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 8ce3ed6	alert, e2e	failed	#5958 (open)	8	e2e (3) ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift 14:50:52 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000e70e20>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-wmkndtkx988t" to finish creating: GET https://rp.j9565440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a9d4baef-549e-4561-b748-439453d113af -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a9d4baef-549e-4561-b748-439453d113af", "name": "a9d4baef-549e-4561-b748-439453d113af", "status": "Failed", "startTime": "2026-08-24T15:49:08.39871397Z", "endTime": "2026-08-24T16:08:11.236941815Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000e70e20>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-wmkndtkx988t" to finish creating: GET https://rp.j9565440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a9d4baef-549e-4561-b748-439453d113af -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a9d4baef-549e-4561-b748-439453d113af", "name": "a9d4baef-549e-4561-b748-439453d113af", "status": "Failed", "startTime": "2026-08-24T15:49:08.39871397Z", "endTime": "2026-08-24T16:08:11.236941815Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift 14:50:52 UTC · failed at e2e · test FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc000408ec0>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-bn9cdfvxcqll" to finish creating: GET https://rp.j9565440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/75726762-be88-402f-a564-60c91125dc8b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/75726762-be88-402f-a564-60c91125dc8b", "name": "75726762-be88-402f-a564-60c91125dc8b", "status": "Failed", "startTime": "2026-08-24T15:47:28.10624954Z", "endTime": "2026-08-24T16:06:31.149377863Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc000408ec0>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-bn9cdfvxcqll" to finish creating: GET https://rp.j9565440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/75726762-be88-402f-a564-60c91125dc8b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/75726762-be88-402f-a564-60c91125dc8b", "name": "75726762-be88-402f-a564-60c91125dc8b", "status": "Failed", "startTime": "2026-08-24T15:47:28.10624954Z", "endTime": "2026-08-24T16:06:31.149377863Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred cluster entered terminal Failed state after role assignment deployment 14:50:52 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.000s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.000s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... alert (5) alert [svc] KubeContainerOOMKilled fired 14:50:52 UTC · failed at alert · test [aro-hcp-observability] [svc] alert KubeContainerOOMKilled does not fire · suite aro-hcp-tests Show raw failure alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-08-24T16:04:45Z Severity: Sev3 Labels: alertname="KubeContainerOOMKilled", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="fluentbit", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="arobit", pod="arobit-forwarder-cxwt8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="OOMKilled", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e03984bb-ef95-4df0-aec9-95c39b0e3c41" Description: Container fluentbit in pod arobit/arobit-forwarder-cxwt8 on cluster ci01-j9565440-mgmt-2 has been OOMKilled. This indicates the container exceeded its memory limit and was terminated by the kernel. Firing 2: State: Fired (not resolved) Started: 2026-08-24T16:04:46Z Severity: Sev3 Labels: alertname="KubeContainerOOMKilled", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="fluentbit", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="arobit", pod="arobit-forwarder-jbtcb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="OOMKilled", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="1ec91183-7c07-473b-b1d1-9df325a61c81" Description: Container fluentbit in pod arobit/arobit-forwarder-jbtcb on cluster ci01-j9565440-mgmt-2 has been OOMKilled. This indicates the container exceeded its memory limit and was terminated by the kernel. Firing 3: State: Fired (not resolved) Started: 2026-08-24T16:04:46Z Severity: Sev3 Labels: alertname="KubeContainerOOMKilled", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="fluentbit", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="arobit", pod="arobit-forwarder-jbtcb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="OOMKilled", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="1ec91183-7c07-473b-b1d1-9df325a61c81" Description: Container fluentbit in pod arobit/arobit-forwarder-jbtcb on cluster ci01-j9565440-mgmt-2 has been OOMKilled. This indicates the container exceeded its memory limit and was terminated by the kernel. Firing 4: State: Fired (not resolved) Started: 2026-08-24T16:04:46Z Severity: Sev3 Labels: alertname="KubeContainerOOMKilled", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="fluentbit", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="arobit", pod="arobit-forwarder-cxwt8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="OOMKilled", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e03984bb-ef95-4df0-aec9-95c39b0e3c41" Description: Container fluentbit in pod arobit/arobit-forwarder-cxwt8 on cluster ci01-j9565440-mgmt-2 has been OOMKilled. This indicates the container exceeded its memory limit and was terminated by the kernel. alert [svc] KubeDaemonSetRolloutStuck fired 14:50:52 UTC · failed at alert · test [aro-hcp-observability] [svc] alert KubeDaemonSetRolloutStuck does not fire · suite aro-hcp-tests Show raw failure alert fired 24 time(s) Firing 1: State: Resolved Started: 2026-08-24T16:11:08Z Ended: 2026-08-24T16:24:07Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-csi-driver", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet kube-system/aks-secrets-store-csi-driver has not finished or progressed for at least 15 minutes. Firing 2: State: Resolved Started: 2026-08-24T16:12:09Z Ended: 2026-08-24T16:24:06Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-csi-driver", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet kube-system/aks-secrets-store-csi-driver has not finished or progressed for at least 15 minutes. Firing 3: State: Resolved Started: 2026-08-24T16:12:09Z Ended: 2026-08-24T16:24:06Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="kube-proxy", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet kube-system/kube-proxy has not finished or progressed for at least 15 minutes. Firing 4: State: Resolved Started: 2026-08-24T16:12:09Z Ended: 2026-08-24T16:24:06Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="cloud-node-manager", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet kube-system/cloud-node-manager has not finished or progressed for at least 15 minutes. Firing 5: State: Resolved Started: 2026-08-24T16:12:09Z Ended: 2026-08-24T16:24:07Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="csi-azuredisk-node", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet kube-system/csi-azuredisk-node has not finished or progressed for at least 15 minutes. Firing 6: State: Resolved Started: 2026-08-24T16:12:09Z Ended: 2026-08-24T16:24:06Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-cns", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet kube-system/azure-cns has not finished or progressed for at least 15 minutes. Firing 7: State: Resolved Started: 2026-08-24T16:12:09Z Ended: 2026-08-24T16:25:05Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="retina-agent", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet kube-system/retina-agent has not finished or progressed for at least 15 minutes. Firing 8: State: Resolved Started: 2026-08-24T16:12:09Z Ended: 2026-08-24T16:24:06Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="ama-metrics-node", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet kube-system/ama-metrics-node has not finished or progressed for at least 15 minutes. Firing 9: State: Resolved Started: 2026-08-24T16:12:09Z Ended: 2026-08-24T16:24:07Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="csi-azurefile-node", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet kube-system/csi-azurefile-node has not finished or progressed for at least 15 minutes. Firing 10: State: Resolved Started: 2026-08-24T16:12:09Z Ended: 2026-08-24T16:24:06Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-provider-azure", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet kube-system/aks-secrets-store-provider-azure has not finished or progressed for at least 15 minutes. Firing 11: State: Resolved Started: 2026-08-24T16:12:09Z Ended: 2026-08-24T16:24:07Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-npm", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet kube-system/azure-npm has not finished or progressed for at least 15 minutes. Firing 12: State: Resolved Started: 2026-08-24T16:13:07Z Ended: 2026-08-24T16:24:07Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-provider-azure", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet kube-system/aks-secrets-store-provider-azure has not finished or progressed for at least 15 minutes. Firing 13: State: Resolved Started: 2026-08-24T16:13:07Z Ended: 2026-08-24T16:24:07Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="csi-azurefile-node", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet kube-system/csi-azurefile-node has not finished or progressed for at least 15 minutes. Firing 14: State: Resolved Started: 2026-08-24T16:13:10Z Ended: 2026-08-24T16:24:06Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-cns", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet kube-system/azure-cns has not finished or progressed for at least 15 minutes. Firing 15: State: Resolved Started: 2026-08-24T16:13:10Z Ended: 2026-08-24T16:24:07Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="csi-azuredisk-node", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet kube-system/csi-azuredisk-node has not finished or progressed for at least 15 minutes. Firing 16: State: Resolved Started: 2026-08-24T16:13:10Z Ended: 2026-08-24T16:24:07Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="kube-proxy", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet kube-system/kube-proxy has not finished or progressed for at least 15 minutes. Firing 17: State: Resolved Started: 2026-08-24T16:13:10Z Ended: 2026-08-24T16:26:07Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="retina-agent", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet kube-system/retina-agent has not finished or progressed for at least 15 minutes. Firing 18: State: Resolved Started: 2026-08-24T16:13:10Z Ended: 2026-08-24T16:24:07Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-npm", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet kube-system/azure-npm has not finished or progressed for at least 15 minutes. Firing 19: State: Resolved Started: 2026-08-24T16:13:10Z Ended: 2026-08-24T16:24:06Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="ama-metrics-node", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet kube-system/ama-metrics-node has not finished or progressed for at least 15 minutes. Firing 20: State: Resolved Started: 2026-08-24T16:13:10Z Ended: 2026-08-24T16:24:07Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="cloud-node-manager", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet kube-system/cloud-node-manager has not finished or progressed for at least 15 minutes. Firing 21: State: Resolved Started: 2026-08-24T16:17:09Z Ended: 2026-08-24T16:24:07Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes. Firing 22: State: Resolved Started: 2026-08-24T16:17:09Z Ended: 2026-08-24T16:23:05Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="node-agent", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="velero", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet velero/node-agent has not finished or progressed for at least 15 minutes. Firing 23: State: Resolved Started: 2026-08-24T16:17:09Z Ended: 2026-08-24T16:24:07Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="node-agent", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="velero", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet velero/node-agent has not finished or progressed for at least 15 minutes. Firing 24: State: Resolved Started: 2026-08-24T16:17:09Z Ended: 2026-08-24T16:29:07Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes. alert [svc] KubePodCrashLooping fired 14:50:52 UTC · failed at alert · test [aro-hcp-observability] [svc] alert KubePodCrashLooping does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-08-24T16:19:07Z Ended: 2026-08-24T16:30:05Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="fluentbit", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="arobit", pod="arobit-forwarder-cxwt8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e03984bb-ef95-4df0-aec9-95c39b0e3c41" Description: Pod arobit/arobit-forwarder-cxwt8 (fluentbit) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Resolved Started: 2026-08-24T16:20:06Z Ended: 2026-08-24T16:33:06Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="fluentbit", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="arobit", pod="arobit-forwarder-cxwt8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e03984bb-ef95-4df0-aec9-95c39b0e3c41" Description: Pod arobit/arobit-forwarder-cxwt8 (fluentbit) is in waiting state (reason: "CrashLoopBackOff"). alert [svc] KubeNodeUnreachable fired 14:50:52 UTC · failed at alert · test [aro-hcp-observability] [svc] alert KubeNodeUnreachable does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-08-24T16:12:59Z Ended: 2026-08-24T16:22:56Z Severity: Sev3 Labels: alertname="KubeNodeUnreachable", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", effect="NoSchedule", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", key="node.kubernetes.io/unreachable", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="prometheus", node="aks-userswft2-20578738-vmss000002", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: aks-userswft2-20578738-vmss000002 is unreachable and some workloads may be rescheduled. Firing 2: State: Resolved Started: 2026-08-24T16:12:59Z Ended: 2026-08-24T16:23:55Z Severity: Sev3 Labels: alertname="KubeNodeUnreachable", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", effect="NoSchedule", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", key="node.kubernetes.io/unreachable", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="prometheus", node="aks-userswft2-20578738-vmss000002", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: aks-userswft2-20578738-vmss000002 is unreachable and some workloads may be rescheduled. alert [svc] KubeDaemonSetMisScheduled fired 14:50:52 UTC · failed at alert · test [aro-hcp-observability] [svc] alert KubeDaemonSetMisScheduled does not fire · suite aro-hcp-tests Show raw failure alert fired 4 time(s) Firing 1: State: Resolved Started: 2026-08-24T16:12:09Z Ended: 2026-08-24T16:24:07Z Severity: Sev3 Labels: alertname="KubeDaemonSetMisScheduled", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="node-agent", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="velero", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: 1 Pods of DaemonSet velero/node-agent are running where they are not supposed to run. Firing 2: State: Resolved Started: 2026-08-24T16:12:09Z Ended: 2026-08-24T16:24:07Z Severity: Sev3 Labels: alertname="KubeDaemonSetMisScheduled", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: 1 Pods of DaemonSet arobit/arobit-forwarder are running where they are not supposed to run. Firing 3: State: Resolved Started: 2026-08-24T16:13:10Z Ended: 2026-08-24T16:23:08Z Severity: Sev3 Labels: alertname="KubeDaemonSetMisScheduled", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: 1 Pods of DaemonSet arobit/arobit-forwarder are running where they are not supposed to run. Firing 4: State: Resolved Started: 2026-08-24T16:13:11Z Ended: 2026-08-24T16:23:08Z Severity: Sev3 Labels: alertname="KubeDaemonSetMisScheduled", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="node-agent", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="velero", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: 1 Pods of DaemonSet velero/node-agent are running where they are not supposed to run.
14:44:27	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c6d4678	alert, e2e	failed	★#6629 (open)	4	e2e (3) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 14:44:27 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc0015a4b20>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-pdzrhq4kn5dh" to finish creating: GET https://rp.j9446912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3839a4cf-1d74-435a-ad7d-cf4ad8e2f726 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3839a4cf-1d74-435a-ad7d-cf4ad8e2f726", "name": "3839a4cf-1d74-435a-ad7d-cf4ad8e2f726", "status": "Failed", "startTime": "2026-08-24T15:35:46.321920583Z", "endTime": "2026-08-24T15:54:46.656362714Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc0015a4b20>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-pdzrhq4kn5dh" to finish creating: GET https://rp.j9446912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3839a4cf-1d74-435a-ad7d-cf4ad8e2f726 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3839a4cf-1d74-435a-ad7d-cf4ad8e2f726", "name": "3839a4cf-1d74-435a-ad7d-cf4ad8e2f726", "status": "Failed", "startTime": "2026-08-24T15:35:46.321920583Z", "endTime": "2026-08-24T15:54:46.656362714Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred cluster entered terminal Failed state after role assignment deployment 14:44:27 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.002s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.002s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 14:44:27 UTC · failed at e2e · test FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc001ab4ac0>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-6b45927pz45x" to finish creating: GET https://rp.j9446912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f98ab769-8943-4e16-9ba2-6234424afc6d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f98ab769-8943-4e16-9ba2-6234424afc6d", "name": "f98ab769-8943-4e16-9ba2-6234424afc6d", "status": "Failed", "startTime": "2026-08-24T15:35:32.53710884Z", "endTime": "2026-08-24T15:54:36.487936419Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc001ab4ac0>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-6b45927pz45x" to finish creating: GET https://rp.j9446912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f98ab769-8943-4e16-9ba2-6234424afc6d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f98ab769-8943-4e16-9ba2-6234424afc6d", "name": "f98ab769-8943-4e16-9ba2-6234424afc6d", "status": "Failed", "startTime": "2026-08-24T15:35:32.53710884Z", "endTime": "2026-08-24T15:54:36.487936419Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 14:44:27 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 7 time(s) (1 unknown, 6 known) Firing 1: State: Resolved Started: 2026-08-24T15:48:52Z Ended: 2026-08-24T16:05:50Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9446912-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd1dkicrfebg5gk7mjo71hbgeofv807-s9h5a0t6o1q6t2p", pod="router-6dd6c76467-mz8vb", severity="warning" Description: Pod ocm-arohcpci01-2sd1dkicrfebg5gk7mjo71hbgeofv807-s9h5a0t6o1q6t2p/router-6dd6c76467-mz8vb has been in a non-ready state for longer than 5 minutes.
14:42:21	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 3ec9ecd	alert, e2e	failed	#6364 (open)	5	e2e (4) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 14:42:21 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:89]: failed to create HCP cluster "arm64-vm-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc00182e2a0>: failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-k4d2bsmbhrnc" to finish creating: GET https://rp.j8539776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d3eb4e8a-5daa-40ba-be6e-b23c10b4eafe -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d3eb4e8a-5daa-40ba-be6e-b23c10b4eafe", "name": "d3eb4e8a-5daa-40ba-be6e-b23c10b4eafe", "status": "Failed", "startTime": "2026-08-24T15:41:38.68937252Z", "endTime": "2026-08-24T16:00:39.082023736Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:89]: failed to create HCP cluster "arm64-vm-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc00182e2a0>: failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-k4d2bsmbhrnc" to finish creating: GET https://rp.j8539776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d3eb4e8a-5daa-40ba-be6e-b23c10b4eafe -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d3eb4e8a-5daa-40ba-be6e-b23c10b4eafe", "name": "d3eb4e8a-5daa-40ba-be6e-b23c10b4eafe", "status": "Failed", "startTime": "2026-08-24T15:41:38.68937252Z", "endTime": "2026-08-24T16:00:39.082023736Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 14:42:21 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:726]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000dd0320>: failed to create HCP cluster np-dg-minor-4tds4q: failed waiting for cluster="np-dg-minor-4tds4q" in resourcegroup="rg-np-dg-minor-4tds4q-6qtw9l2fwcdn" to finish creating: GET https://rp.j8539776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/162a10b5-886b-40a8-a95b-30fc38ce9bc5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/162a10b5-886b-40a8-a95b-30fc38ce9bc5", "name": "162a10b5-886b-40a8-a95b-30fc38ce9bc5", "status": "Failed", "startTime": "2026-08-24T15:41:41.593852701Z", "endTime": "2026-08-24T16:00:49.106385261Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:726]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000dd0320>: failed to create HCP cluster np-dg-minor-4tds4q: failed waiting for cluster="np-dg-minor-4tds4q" in resourcegroup="rg-np-dg-minor-4tds4q-6qtw9l2fwcdn" to finish creating: GET https://rp.j8539776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/162a10b5-886b-40a8-a95b-30fc38ce9bc5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/162a10b5-886b-40a8-a95b-30fc38ce9bc5", "name": "162a10b5-886b-40a8-a95b-30fc38ce9bc5", "status": "Failed", "startTime": "2026-08-24T15:41:41.593852701Z", "endTime": "2026-08-24T16:00:49.106385261Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 14:42:21 UTC · failed at e2e · test SRE should be able to retrieve serial console logs for a VM · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:310]: failed to create HCP cluster "sre-hcp-cluster-sc" for serial console test Unexpected error: <fmt.wrapError \| 0xc00127e240>: failed to create HCP cluster sre-hcp-cluster-sc: failed waiting for cluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-4s95f76fslkw" to finish creating: GET https://rp.j8539776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a5bec99d-ae1e-4543-a8ff-8aa060972e34 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a5bec99d-ae1e-4543-a8ff-8aa060972e34", "name": "a5bec99d-ae1e-4543-a8ff-8aa060972e34", "status": "Failed", "startTime": "2026-08-24T15:41:58.968228589Z", "endTime": "2026-08-24T16:00:59.046704933Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:310]: failed to create HCP cluster "sre-hcp-cluster-sc" for serial console test Unexpected error: <fmt.wrapError \| 0xc00127e240>: failed to create HCP cluster sre-hcp-cluster-sc: failed waiting for cluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-4s95f76fslkw" to finish creating: GET https://rp.j8539776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a5bec99d-ae1e-4543-a8ff-8aa060972e34 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a5bec99d-ae1e-4543-a8ff-8aa060972e34", "name": "a5bec99d-ae1e-4543-a8ff-8aa060972e34", "status": "Failed", "startTime": "2026-08-24T15:41:58.968228589Z", "endTime": "2026-08-24T16:00:59.046704933Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift 14:42:21 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:109]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling Unexpected error: <fmt.wrapError \| 0xc000fa81a0>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-jhvt8ptm7fw8" to finish creating: GET https://rp.j8539776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/12485de2-db77-439f-89df-5bacb5d218b2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/12485de2-db77-439f-89df-5bacb5d218b2", "name": "12485de2-db77-439f-89df-5bacb5d218b2", "status": "Failed", "startTime": "2026-08-24T15:41:38.67903036Z", "endTime": "2026-08-24T16:00:39.109568382Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:109]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling Unexpected error: <fmt.wrapError \| 0xc000fa81a0>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-jhvt8ptm7fw8" to finish creating: GET https://rp.j8539776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/12485de2-db77-439f-89df-5bacb5d218b2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/12485de2-db77-439f-89df-5bacb5d218b2", "name": "12485de2-db77-439f-89df-5bacb5d218b2", "status": "Failed", "startTime": "2026-08-24T15:41:38.67903036Z", "endTime": "2026-08-24T16:00:39.109568382Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 14:42:21 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 3 time(s) (2 unknown, 1 known) Firing 1: State: Resolved Started: 2026-08-24T15:55:28Z Ended: 2026-08-24T16:09:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8539776-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd1ghoa165uh75qt39elkm018udft1l-u7o1r1e3a1i8p2h", pod="router-bc5b66df4-m2tkb", severity="warning" Description: Pod ocm-arohcpci01-2sd1ghoa165uh75qt39elkm018udft1l-u7o1r1e3a1i8p2h/router-bc5b66df4-m2tkb has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-08-24T15:56:26Z Ended: 2026-08-24T16:09:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8539776-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd1gcvqlv2t9s69eq6f5mdfd7qjbb5f-h8k0a3a7f2f5p1y", pod="router-7b6dfdd9b-2c6nj", severity="warning" Description: Pod ocm-arohcpci01-2sd1gcvqlv2t9s69eq6f5mdfd7qjbb5f-h8k0a3a7f2f5p1y/router-7b6dfdd9b-2c6nj has been in a non-ready state for longer than 5 minutes.
14:36:42	pull-ci-Azure-ARO-HCP-main-e2e-parallel head ec049a8	n/a	passed	#6643 (open)	0	n/a
14:34:02	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 54bb2db	alert, e2e	failed	#6575 (open)	7	e2e (6) ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift 14:34:02 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test Unexpected error: <fmt.wrapError \| 0xc000e86920>: failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-7vpnvrbqnldf" to finish creating: GET https://rp.j1362816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/73cc6203-30af-4c53-9024-398221683c83 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/73cc6203-30af-4c53-9024-398221683c83", "name": "73cc6203-30af-4c53-9024-398221683c83", "status": "Failed", "startTime": "2026-08-24T15:17:58.555796542Z", "endTime": "2026-08-24T15:37:08.01693151Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test Unexpected error: <fmt.wrapError \| 0xc000e86920>: failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-7vpnvrbqnldf" to finish creating: GET https://rp.j1362816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/73cc6203-30af-4c53-9024-398221683c83 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/73cc6203-30af-4c53-9024-398221683c83", "name": "73cc6203-30af-4c53-9024-398221683c83", "status": "Failed", "startTime": "2026-08-24T15:17:58.555796542Z", "endTime": "2026-08-24T15:37:08.01693151Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 14:34:02 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc000a808e0>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-cgt6b2gspgmg" to finish creating: GET https://rp.j1362816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c64a33f9-1d34-4ca6-89af-ad50a67745b6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c64a33f9-1d34-4ca6-89af-ad50a67745b6", "name": "c64a33f9-1d34-4ca6-89af-ad50a67745b6", "status": "Failed", "startTime": "2026-08-24T15:19:10.873312133Z", "endTime": "2026-08-24T15:38:17.935941947Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, cluster-node-tuning-operator, redhat-marketplace-catalog, cluster-network-operator, redhat-operators-catalog, catalog-operator, openshift-route-controller-manager, ingress-operator, hosted-cluster-config-operator, packageserver, cluster-policy-controller, cluster-storage-operator, certified-operators-catalog, olm-operator, openshift-controller-manager, csi-snapshot-controller-operator, community-operators-catalog, dns-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc000a808e0>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-cgt6b2gspgmg" to finish creating: GET https://rp.j1362816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c64a33f9-1d34-4ca6-89af-ad50a67745b6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c64a33f9-1d34-4ca6-89af-ad50a67745b6", "name": "c64a33f9-1d34-4ca6-89af-ad50a67745b6", "status": "Failed", "startTime": "2026-08-24T15:19:10.873312133Z", "endTime": "2026-08-24T15:38:17.935941947Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, cluster-node-tuning-operator, redhat-marketplace-catalog, cluster-network-operator, redhat-operators-catalog, catalog-operator, openshift-route-controller-manager, ingress-operator, hosted-cluster-config-operator, packageserver, cluster-policy-controller, cluster-storage-operator, certified-operators-catalog, olm-operator, openshift-controller-manager, csi-snapshot-controller-operator, community-operators-catalog, dns-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift 14:34:02 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260630preview and verify the ingress is internal · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc001580bc0>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-cbbn9kpl7fxv" to finish creating: GET https://rp.j1362816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c6f112da-48e3-4cbb-a1b9-a47159e85ff5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c6f112da-48e3-4cbb-a1b9-a47159e85ff5", "name": "c6f112da-48e3-4cbb-a1b9-a47159e85ff5", "status": "Failed", "startTime": "2026-08-24T15:18:01.504032647Z", "endTime": "2026-08-24T15:37:08.017891025Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, azure-cloud-controller-manager, cluster-node-tuning-operator, olm-operator, olm-collect-profiles, cluster-autoscaler, control-plane-pki-operator, kube-controller-manager, community-operators-catalog, redhat-operators-catalog, machine-approver, openshift-controller-manager, cluster-storage-operator, dns-operator, redhat-marketplace-catalog, cluster-version-operator, openshift-route-controller-manager, ingress-operator, kube-scheduler, cluster-image-registry-operator, konnectivity-agent, catalog-operator, cluster-network-operator, ignition-server, ignition-server-proxy, openshift-apiserver, hosted-cluster-config-operator, csi-snapshot-controller-operator, packageserver, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc001580bc0>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-cbbn9kpl7fxv" to finish creating: GET https://rp.j1362816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c6f112da-48e3-4cbb-a1b9-a47159e85ff5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c6f112da-48e3-4cbb-a1b9-a47159e85ff5", "name": "c6f112da-48e3-4cbb-a1b9-a47159e85ff5", "status": "Failed", "startTime": "2026-08-24T15:18:01.504032647Z", "endTime": "2026-08-24T15:37:08.017891025Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, azure-cloud-controller-manager, cluster-node-tuning-operator, olm-operator, olm-collect-profiles, cluster-autoscaler, control-plane-pki-operator, kube-controller-manager, community-operators-catalog, redhat-operators-catalog, machine-approver, openshift-controller-manager, cluster-storage-operator, dns-operator, redhat-marketplace-catalog, cluster-version-operator, openshift-route-controller-manager, ingress-operator, kube-scheduler, cluster-image-registry-operator, konnectivity-agent, catalog-operator, cluster-network-operator, ignition-server, ignition-server-proxy, openshift-apiserver, hosted-cluster-config-operator, csi-snapshot-controller-operator, packageserver, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 14:34:02 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0xc000e96560>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-fn2m5nmfttf7" to finish creating: GET https://rp.j1362816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eb7b7e77-c07f-48f4-a1bf-2f48bcb6434a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eb7b7e77-c07f-48f4-a1bf-2f48bcb6434a", "name": "eb7b7e77-c07f-48f4-a1bf-2f48bcb6434a", "status": "Failed", "startTime": "2026-08-24T15:17:08.176674236Z", "endTime": "2026-08-24T15:36:17.96458443Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0xc000e96560>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-fn2m5nmfttf7" to finish creating: GET https://rp.j1362816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eb7b7e77-c07f-48f4-a1bf-2f48bcb6434a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eb7b7e77-c07f-48f4-a1bf-2f48bcb6434a", "name": "eb7b7e77-c07f-48f4-a1bf-2f48bcb6434a", "status": "Failed", "startTime": "2026-08-24T15:17:08.176674236Z", "endTime": "2026-08-24T15:36:17.96458443Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator, ignition-server-proxy, kube-controller-manager, router]; provider Microsoft.RedHatOpenShift 14:34:02 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--wkl8qfr9l4cs/cluster-hs-pre-jtp7jx should provision Unexpected error: <fmt.wrapError \| 0xc0009c0960>: failed to create HCP cluster cluster-hs-pre-jtp7jx: failed waiting for cluster="cluster-hs-pre-jtp7jx" in resourcegroup="rg-hs-pre--wkl8qfr9l4cs" to finish creating: GET https://rp.j1362816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/df253c01-fd54-4465-9725-47d0942d268e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/df253c01-fd54-4465-9725-47d0942d268e", "name": "df253c01-fd54-4465-9725-47d0942d268e", "status": "Failed", "startTime": "2026-08-24T15:18:48.695053949Z", "endTime": "2026-08-24T15:37:57.993163664Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--wkl8qfr9l4cs/cluster-hs-pre-jtp7jx should provision Unexpected error: <fmt.wrapError \| 0xc0009c0960>: failed to create HCP cluster cluster-hs-pre-jtp7jx: failed waiting for cluster="cluster-hs-pre-jtp7jx" in resourcegroup="rg-hs-pre--wkl8qfr9l4cs" to finish creating: GET https://rp.j1362816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/df253c01-fd54-4465-9725-47d0942d268e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/df253c01-fd54-4465-9725-47d0942d268e", "name": "df253c01-fd54-4465-9725-47d0942d268e", "status": "Failed", "startTime": "2026-08-24T15:18:48.695053949Z", "endTime": "2026-08-24T15:37:57.993163664Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 14:34:02 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:599]: failed to create nodepool npdg-4-21 Unexpected error: <fmt.wrapError \| 0xc000e96960>: failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-q2tthc" in resourcegroup="rg-np-downgrade-q2tthc-z7rbl7r2jwws" to finish creating: GET https://rp.j1362816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/47831462-3285-4a15-8722-8601b5708cf5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/47831462-3285-4a15-8722-8601b5708cf5", "name": "47831462-3285-4a15-8722-8601b5708cf5", "status": "Failed", "startTime": "2026-08-24T15:29:26.286804415Z", "endTime": "2026-08-24T15:48:34.685207308Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:599]: failed to create nodepool npdg-4-21 Unexpected error: <fmt.wrapError \| 0xc000e96960>: failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-q2tthc" in resourcegroup="rg-np-downgrade-q2tthc-z7rbl7r2jwws" to finish creating: GET https://rp.j1362816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/47831462-3285-4a15-8722-8601b5708cf5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/47831462-3285-4a15-8722-8601b5708cf5", "name": "47831462-3285-4a15-8722-8601b5708cf5", "status": "Failed", "startTime": "2026-08-24T15:29:26.286804415Z", "endTime": "2026-08-24T15:48:34.685207308Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 14:34:02 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 14 time(s) (4 unknown, 10 known) Firing 1: State: Resolved Started: 2026-08-24T15:32:32Z Ended: 2026-08-24T15:39:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1362816-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd15mfbnh569osr6v3jpmrh2u4ovste-n1x0n6k7s4n4s4m", pod="etcd-0", severity="warning" Description: Pod ocm-arohcpci01-2sd15mfbnh569osr6v3jpmrh2u4ovste-n1x0n6k7s4n4s4m/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-08-24T15:33:32Z Ended: 2026-08-24T15:39:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1362816-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd15s2tbl2mk3taqpkr4l2ue96f7dbj-agg-cluster", pod="cluster-api-7b8d6c548b-n9txh", severity="warning" Description: Pod ocm-arohcpci01-2sd15s2tbl2mk3taqpkr4l2ue96f7dbj-agg-cluster/cluster-api-7b8d6c548b-n9txh has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-08-24T15:33:32Z Ended: 2026-08-24T15:39:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1362816-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd16iscun46vl67eantdmfbrchq5cvk-q4u5f1h8q8y4t5a", pod="kube-state-metrics-hcp-5b868d7ddd-fhcvs", severity="warning" Description: Pod ocm-arohcpci01-2sd16iscun46vl67eantdmfbrchq5cvk-q4u5f1h8q8y4t5a/kube-state-metrics-hcp-5b868d7ddd-fhcvs has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-08-24T15:33:32Z Ended: 2026-08-24T15:39:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1362816-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd15s2tbl2mk3taqpkr4l2ue96f7dbj-agg-cluster", pod="control-plane-operator-6dffd99944-b2sdg", severity="warning" Description: Pod ocm-arohcpci01-2sd15s2tbl2mk3taqpkr4l2ue96f7dbj-agg-cluster/control-plane-operator-6dffd99944-b2sdg has been in a non-ready state for longer than 5 minutes.
14:29:42	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 0633420	unknown	failed	#6575 (open)	1	non-artifact-backed failure (no junit artifacts)
14:28:36	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 9906d38	e2e	failed	#6658 (open)	4	e2e (4) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 14:28:36 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc0008f0180>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-fvnthdb9mrqx" to finish creating: GET https://rp.j2797056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3adf48b6-e1aa-4b6a-93ab-0fddaf491e0c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3adf48b6-e1aa-4b6a-93ab-0fddaf491e0c", "name": "3adf48b6-e1aa-4b6a-93ab-0fddaf491e0c", "status": "Failed", "startTime": "2026-08-24T15:18:10.302856594Z", "endTime": "2026-08-24T15:37:11.239396006Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc0008f0180>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-fvnthdb9mrqx" to finish creating: GET https://rp.j2797056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3adf48b6-e1aa-4b6a-93ab-0fddaf491e0c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3adf48b6-e1aa-4b6a-93ab-0fddaf491e0c", "name": "3adf48b6-e1aa-4b6a-93ab-0fddaf491e0c", "status": "Failed", "startTime": "2026-08-24T15:18:10.302856594Z", "endTime": "2026-08-24T15:37:11.239396006Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred Cluster provisioning failed 14:28:36 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift 14:28:36 UTC · failed at e2e · test SRE can pause schedules to stop backup execution for an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc001078580>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-96jvrlw87dhc" to finish creating: GET https://rp.j2797056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f52090fa-71d3-447f-ad30-f731526cbb4b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f52090fa-71d3-447f-ad30-f731526cbb4b", "name": "f52090fa-71d3-447f-ad30-f731526cbb4b", "status": "Failed", "startTime": "2026-08-24T15:16:10.209370894Z", "endTime": "2026-08-24T15:35:11.305317842Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc001078580>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-96jvrlw87dhc" to finish creating: GET https://rp.j2797056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f52090fa-71d3-447f-ad30-f731526cbb4b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f52090fa-71d3-447f-ad30-f731526cbb4b", "name": "f52090fa-71d3-447f-ad30-f731526cbb4b", "status": "Failed", "startTime": "2026-08-24T15:16:10.209370894Z", "endTime": "2026-08-24T15:35:11.305317842Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 14:28:36 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0xc000698540>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-rqwql22jhw9b" to finish creating: GET https://rp.j2797056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9120d0f9-847c-44de-98bf-37d19b45ef2f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9120d0f9-847c-44de-98bf-37d19b45ef2f", "name": "9120d0f9-847c-44de-98bf-37d19b45ef2f", "status": "Failed", "startTime": "2026-08-24T15:16:29.566422332Z", "endTime": "2026-08-24T15:35:31.366369174Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0xc000698540>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-rqwql22jhw9b" to finish creating: GET https://rp.j2797056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9120d0f9-847c-44de-98bf-37d19b45ef2f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9120d0f9-847c-44de-98bf-37d19b45ef2f", "name": "9120d0f9-847c-44de-98bf-37d19b45ef2f", "status": "Failed", "startTime": "2026-08-24T15:16:29.566422332Z", "endTime": "2026-08-24T15:35:31.366369174Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred
14:18:07	pull-ci-Azure-ARO-HCP-main-e2e-parallel head cbb6788	alert, e2e	failed	#6232 (open)	5	e2e (4) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 14:18:07 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc0010deff0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-4cmmkc" in resourcegroup="admin-credential-lifecycle-test-mk5bt6ghv29k" to finish deleting: GET https://rp.j9012096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0644f278-df05-4603-a10d-3a66a1f851d7 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0644f278-df05-4603-a10d-3a66a1f851d7", "name": "0644f278-df05-4603-a10d-3a66a1f851d7", "status": "Failed", "startTime": "2026-08-24T15:17:03.672410989Z", "endTime": "2026-08-24T15:41:10.046088761Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sd0tjfmep0gog9fd4pfkr5bnpc0n642 still exists (deletion dispatched at 2026-08-24T15:17:04Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc0010deff0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-4cmmkc" in resourcegroup="admin-credential-lifecycle-test-mk5bt6ghv29k" to finish deleting: GET https://rp.j9012096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0644f278-df05-4603-a10d-3a66a1f851d7 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0644f278-df05-4603-a10d-3a66a1f851d7", "name": "0644f278-df05-4603-a10d-3a66a1f851d7", "status": "Failed", "startTime": "2026-08-24T15:17:03.672410989Z", "endTime": "2026-08-24T15:41:10.046088761Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sd0tjfmep0gog9fd4pfkr5bnpc0n642 still exists (deletion dispatched at 2026-08-24T15:17:04Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] <no_message>; [clusterServiceStatus] <no_message>; [descendantResources] <no_message>; [hostedCluster] <no_message>; provider Microsoft.RedHatOpenShift 14:18:07 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc001c982a0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-hs-pre-gdnhnw" in resourcegroup="rg-hs-pre--hm7prkdxbkw6" to finish deleting: GET https://rp.j9012096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67ed1acc-0077-4b07-848d-484f46170f44 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67ed1acc-0077-4b07-848d-484f46170f44", "name": "67ed1acc-0077-4b07-848d-484f46170f44", "status": "Failed", "startTime": "2026-08-24T15:21:50.42303027Z", "endTime": "2026-08-24T15:45:59.872804145Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] \u003cno_message\u003e; [clusterServiceStatus] \u003cno_message\u003e; [descendantResources] \u003cno_message\u003e; [hostedCluster] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc001c982a0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-hs-pre-gdnhnw" in resourcegroup="rg-hs-pre--hm7prkdxbkw6" to finish deleting: GET https://rp.j9012096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67ed1acc-0077-4b07-848d-484f46170f44 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67ed1acc-0077-4b07-848d-484f46170f44", "name": "67ed1acc-0077-4b07-848d-484f46170f44", "status": "Failed", "startTime": "2026-08-24T15:21:50.42303027Z", "endTime": "2026-08-24T15:45:59.872804145Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] \u003cno_message\u003e; [clusterServiceStatus] \u003cno_message\u003e; [descendantResources] \u003cno_message\u003e; [hostedCluster] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 14:18:07 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc000e8af48>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-mb2sm98hvbm8" to finish deleting: GET https://rp.j9012096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76928be7-64f8-4a4c-85f6-cd6b790148b9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76928be7-64f8-4a4c-85f6-cd6b790148b9", "name": "76928be7-64f8-4a4c-85f6-cd6b790148b9", "status": "Failed", "startTime": "2026-08-24T15:14:43.826676557Z", "endTime": "2026-08-24T15:38:50.051171369Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sd0tm25j05l3q2f8a1e83cjmn4lda8g still exists (deletion dispatched at 2026-08-24T15:14:45Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc000e8af48>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-mb2sm98hvbm8" to finish deleting: GET https://rp.j9012096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76928be7-64f8-4a4c-85f6-cd6b790148b9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76928be7-64f8-4a4c-85f6-cd6b790148b9", "name": "76928be7-64f8-4a4c-85f6-cd6b790148b9", "status": "Failed", "startTime": "2026-08-24T15:14:43.826676557Z", "endTime": "2026-08-24T15:38:50.051171369Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sd0tm25j05l3q2f8a1e83cjmn4lda8g still exists (deletion dispatched at 2026-08-24T15:14:45Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [descendantResources] remaining resources: nodePools, serviceProviderClusters; provider Microsoft.RedHatOpenShift 14:18:07 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc000ebc2a0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-0-h4ldj6" in resourcegroup="rg-candidate-5-0-xjnldggtrt7m" to finish deleting: GET https://rp.j9012096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fa0102ea-068c-4907-92f2-69579c623cac -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fa0102ea-068c-4907-92f2-69579c623cac", "name": "fa0102ea-068c-4907-92f2-69579c623cac", "status": "Failed", "startTime": "2026-08-24T15:21:51.698922375Z", "endTime": "2026-08-24T15:45:59.987785659Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sd0tm5ibpilvjmdnr77hbqf9t255bon still exists (deletion dispatched at 2026-08-24T15:21:51Z); [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc000ebc2a0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-0-h4ldj6" in resourcegroup="rg-candidate-5-0-xjnldggtrt7m" to finish deleting: GET https://rp.j9012096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fa0102ea-068c-4907-92f2-69579c623cac -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fa0102ea-068c-4907-92f2-69579c623cac", "name": "fa0102ea-068c-4907-92f2-69579c623cac", "status": "Failed", "startTime": "2026-08-24T15:21:51.698922375Z", "endTime": "2026-08-24T15:45:59.987785659Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sd0tm5ibpilvjmdnr77hbqf9t255bon still exists (deletion dispatched at 2026-08-24T15:21:51Z); [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 14:18:07 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 12 time(s) (5 unknown, 7 known) Firing 1: State: Resolved Started: 2026-08-24T15:11:48Z Ended: 2026-08-24T15:18:47Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9012096-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd0tivj0u60lm1dth5pjmkuarro34do-n8n9o9a8t2s4e6m", pod="kube-apiserver-795c5dc94f-bggd8", severity="warning" Description: Pod ocm-arohcpci01-2sd0tivj0u60lm1dth5pjmkuarro34do-n8n9o9a8t2s4e6m/kube-apiserver-795c5dc94f-bggd8 has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-08-24T15:11:48Z Ended: 2026-08-24T15:18:48Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9012096-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd0tsdt2doqqre7oevhvs2m0rcviir2-w9k0g8e9q6b2y5a", pod="etcd-1", severity="warning" Description: Pod ocm-arohcpci01-2sd0tsdt2doqqre7oevhvs2m0rcviir2-w9k0g8e9q6b2y5a/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-08-24T15:11:48Z Ended: 2026-08-24T15:18:47Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9012096-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd0tl62srf8l23gsr4t3od82f106gap-m2c3u3c1r4o8d7i", pod="etcd-2", severity="warning" Description: Pod ocm-arohcpci01-2sd0tl62srf8l23gsr4t3od82f106gap-m2c3u3c1r4o8d7i/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-08-24T15:11:48Z Ended: 2026-08-24T15:18:48Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9012096-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd0tcss6f44egtai516gd8nov4k698p-v20260901", pod="etcd-2", severity="warning" Description: Pod ocm-arohcpci01-2sd0tcss6f44egtai516gd8nov4k698p-v20260901/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 5: State: Resolved Started: 2026-08-24T15:11:48Z Ended: 2026-08-24T15:18:48Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9012096-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd0tlmtcpd07kvkm18o0djt9sfj7dcm-n3g9j2h5i2r5y3t", pod="etcd-2", severity="warning" Description: Pod ocm-arohcpci01-2sd0tlmtcpd07kvkm18o0djt9sfj7dcm-n3g9j2h5i2r5y3t/etcd-2 has been in a non-ready state for longer than 5 minutes.
13:56:59	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 0633420	unknown	failed	#6575 (open)	1	non-artifact-backed failure (no junit artifacts)
13:56:30	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 3c81c74	unknown	failed	#6645 (open)	1	non-artifact-backed failure (no junit artifacts)
13:28:36	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 49b4773 · merge 49b4773	n/a	passed	#6585 (merged)	0	n/a
13:27:03	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 3c81c74	unknown	failed	#6645 (open)	1	non-artifact-backed failure (no junit artifacts)
13:27:00	pull-ci-Azure-ARO-HCP-main-e2e-parallel head d31f142	e2e	failed	⚠#6429 (open)	2	e2e (2) run command "e2e-runcommand-1787582391625387873" failed with exit code 1 on VM "<vm>" (output: "Client Version: v1.36.4 Kustomize Version: v5.8.1") 13:27:00 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:160]: kubectl version should succeed from VM via private KAS internal LB (output: ) Unexpected error: <errors.errorString \| 0xc0007fbf60>: run command "e2e-runcommand-1787582391625387873" failed with exit code 1 on VM "private-kas-test-vm" (output: "Client Version: v1.36.4\nKustomize Version: v5.8.1") ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:160]: kubectl version should succeed from VM via private KAS internal LB (output: ) Unexpected error: <errors.errorString \| 0xc0007fbf60>: run command "e2e-runcommand-1787582391625387873" failed with exit code 1 on VM "private-kas-test-vm" (output: "Client Version: v1.36.4\nKustomize Version: v5.8.1") ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 13:27:00 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--k24nwww6fmhh/cluster-hs-pre-zdf4g7 should provision Unexpected error: <fmt.wrapError \| 0xc00111c160>: failed to create HCP cluster cluster-hs-pre-zdf4g7: failed waiting for cluster="cluster-hs-pre-zdf4g7" in resourcegroup="rg-hs-pre--k24nwww6fmhh" to finish creating: GET https://rp.j5696384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a0975238-90c3-4b41-87eb-c874aa41d8d7 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a0975238-90c3-4b41-87eb-c874aa41d8d7", "name": "a0975238-90c3-4b41-87eb-c874aa41d8d7", "status": "Failed", "startTime": "2026-08-24T14:14:48.351398791Z", "endTime": "2026-08-24T14:33:48.407943371Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--k24nwww6fmhh/cluster-hs-pre-zdf4g7 should provision Unexpected error: <fmt.wrapError \| 0xc00111c160>: failed to create HCP cluster cluster-hs-pre-zdf4g7: failed waiting for cluster="cluster-hs-pre-zdf4g7" in resourcegroup="rg-hs-pre--k24nwww6fmhh" to finish creating: GET https://rp.j5696384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a0975238-90c3-4b41-87eb-c874aa41d8d7 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a0975238-90c3-4b41-87eb-c874aa41d8d7", "name": "a0975238-90c3-4b41-87eb-c874aa41d8d7", "status": "Failed", "startTime": "2026-08-24T14:14:48.351398791Z", "endTime": "2026-08-24T14:33:48.407943371Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred
13:26:52	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 9ec6505	alert, e2e	failed	#6433 (open)	2	e2e (1) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 13:26:52 UTC · failed at e2e · test FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc000f24820>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-jdqbgdfs67t9" to finish creating: GET https://rp.j6438784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/733891b5-0d3d-4dee-bbf2-0fd76696850b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/733891b5-0d3d-4dee-bbf2-0fd76696850b", "name": "733891b5-0d3d-4dee-bbf2-0fd76696850b", "status": "Failed", "startTime": "2026-08-24T14:17:00.887821869Z", "endTime": "2026-08-24T14:36:05.25428442Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc000f24820>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-jdqbgdfs67t9" to finish creating: GET https://rp.j6438784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/733891b5-0d3d-4dee-bbf2-0fd76696850b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/733891b5-0d3d-4dee-bbf2-0fd76696850b", "name": "733891b5-0d3d-4dee-bbf2-0fd76696850b", "status": "Failed", "startTime": "2026-08-24T14:17:00.887821869Z", "endTime": "2026-08-24T14:36:05.25428442Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [svc] BackendControllerQueueDepthHigh fired 13:26:52 UTC · failed at alert · test [aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-24T14:40:10Z Ended: 2026-08-24T14:49:11Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j6438784-svc", component="backend", name="csstatedump", severity="warning" Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
13:01:35	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 4cf6086	unknown	failed	#6643 (open)	1	non-artifact-backed failure (no junit artifacts)
12:59:40	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 669f42e	provision	failed	#6618 (open)	1	provision (1) ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute 12:59:40 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite step graph Show raw failure time=2026-08-24T13:26:56.540Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam" time=2026-08-24T13:26:58.762Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster time=2026-08-24T13:27:00.485Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=abeb20b90f6fb4be4d4a69c08ed3bbe8b5be5f3247506ae425e325b85b345969 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j8084480-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fabeb20b90f6fb4be4d4a69c08ed3bbe8b5be5f3247506ae425e325b85b345969 time=2026-08-24T13:35:32.482Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8084480-svc/providers/Microsoft.Resources/deployments/abeb20b90f6fb4be4d4a69c08ed3bbe8b5be5f3247506ae425e325b85b345969/operationStatuses/08584140288659502671\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8084480-svc/providers/Microsoft.Resources/deployments/cluster-kyv757fphd2fm\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8084480-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8084480-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8084480-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j8084480-svc/agentPools/u64d8dsv61\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8084480-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-24652102-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 3, 2), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8084480-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-24652102-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
12:58:16	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 515e34b	provision	failed	#6583 (open)	1	provision (1) ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute 12:58:16 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster (stamp=2) · suite step graph Show raw failure time=2026-08-24T13:31:41.774Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 description="Step cluster\n Kind: ARM\n Template: templates/mgmt-cluster.bicep\n Parameters: configurations/mgmt-cluster.tmpl.bicepparam" time=2026-08-24T13:31:42.863Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 time=2026-08-24T13:31:44.020Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 deployment=cf8b4dcb00b286eb41bc03071739c50d179926b7125e9927b43e768ae20b10a5 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j5557248-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fcf8b4dcb00b286eb41bc03071739c50d179926b7125e9927b43e768ae20b10a5 time=2026-08-24T13:39:15.842Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5557248-mgmt-2/providers/Microsoft.Resources/deployments/cf8b4dcb00b286eb41bc03071739c50d179926b7125e9927b43e768ae20b10a5/operationStatuses/08584140285820210806\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5557248-mgmt-2/providers/Microsoft.Resources/deployments/cluster-ce5kkcjcnxbu2\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5557248-mgmt-2/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5557248-mgmt-2/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5557248-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j5557248-mgmt-2/agentPools/infra1\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5557248-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-20013346-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 3, 2), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5557248-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-20013346-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
12:58:09	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 0633420	provision	failed	#6575 (open)	1	provision (1) ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute 12:58:09 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite step graph Show raw failure time=2026-08-24T13:26:28.921Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam" time=2026-08-24T13:26:30.957Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster time=2026-08-24T13:26:32.602Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=9508f110ddf9b49e79a1c674ffa6223d05182dbd61375e2ba689c04c8113e848 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2132480-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F9508f110ddf9b49e79a1c674ffa6223d05182dbd61375e2ba689c04c8113e848 time=2026-08-24T13:34:04.585Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2132480-svc/providers/Microsoft.Resources/deployments/9508f110ddf9b49e79a1c674ffa6223d05182dbd61375e2ba689c04c8113e848/operationStatuses/08584140288938416315\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2132480-svc/providers/Microsoft.Resources/deployments/cluster-d54rkmqgu5xlq\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2132480-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2132480-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2132480-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j2132480-svc/agentPools/u64d8dsv61\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2132480-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-42034125-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 3, 2), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2132480-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-42034125-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
12:58:03	pull-ci-Azure-ARO-HCP-main-e2e-parallel head af0ab1f	provision	failed	#6504 (open)	1	provision (1) ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute 12:58:03 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster (stamp=1) · suite step graph Show raw failure time=2026-08-24T13:31:05.820Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 description="Step cluster\n Kind: ARM\n Template: templates/mgmt-cluster.bicep\n Parameters: configurations/mgmt-cluster.tmpl.bicepparam" time=2026-08-24T13:31:07.064Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 time=2026-08-24T13:31:08.065Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 deployment=ed1457ef8895351af8e0cf66909d3e8cde536aaa7acb6606665b7650f6a24978 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j7970432-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fed1457ef8895351af8e0cf66909d3e8cde536aaa7acb6606665b7650f6a24978 time=2026-08-24T13:39:10.218Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7970432-mgmt-1/providers/Microsoft.Resources/deployments/ed1457ef8895351af8e0cf66909d3e8cde536aaa7acb6606665b7650f6a24978/operationStatuses/08584140286179722113\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7970432-mgmt-1/providers/Microsoft.Resources/deployments/cluster-bydjouzz4qrms\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7970432-mgmt-1/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7970432-mgmt-1/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7970432-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j7970432-mgmt-1/agentPools/infra1\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7970432-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-20554033-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 3, 2), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7970432-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-20554033-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
12:39:55	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 49b4773 · merge 49b4773	provision	failed	#6585 (merged)	1	provision (1) ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute 12:39:55 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite step graph Show raw failure time=2026-08-24T13:06:20.681Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam" time=2026-08-24T13:06:22.594Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster time=2026-08-24T13:06:24.444Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=03ed80b7fb9f53548d372e1053a4aca53aea99bb193680d269ceba2e4aa413ec portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j6085120-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F03ed80b7fb9f53548d372e1053a4aca53aea99bb193680d269ceba2e4aa413ec time=2026-08-24T13:14:56.464Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6085120-svc/providers/Microsoft.Resources/deployments/03ed80b7fb9f53548d372e1053a4aca53aea99bb193680d269ceba2e4aa413ec/operationStatuses/08584140301019888549\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6085120-svc/providers/Microsoft.Resources/deployments/cluster-mw4vmkbkun3py\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6085120-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6085120-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6085120-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j6085120-svc/agentPools/u64d8dsv61\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6085120-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-45523988-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 3, 2), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6085120-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-45523988-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
12:36:32	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 49b4773 · merge 49b4773	provision	failed	#6585 (merged)	1	provision (1) ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute 12:36:32 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster (stamp=1) · suite step graph Show raw failure time=2026-08-24T13:04:34.083Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 description="Step cluster\n Kind: ARM\n Template: templates/mgmt-cluster.bicep\n Parameters: configurations/mgmt-cluster.tmpl.bicepparam" time=2026-08-24T13:04:35.141Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 time=2026-08-24T13:04:36.377Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 deployment=4989b67b02de49f77c8217e80b11221be6396694deee2c6d1f0f2599c72fa63b portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j4136192-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F4989b67b02de49f77c8217e80b11221be6396694deee2c6d1f0f2599c72fa63b time=2026-08-24T13:12:38.165Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4136192-mgmt-1/providers/Microsoft.Resources/deployments/4989b67b02de49f77c8217e80b11221be6396694deee2c6d1f0f2599c72fa63b/operationStatuses/08584140302097177966\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4136192-mgmt-1/providers/Microsoft.Resources/deployments/cluster-n72exdzvr6uha\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4136192-mgmt-1/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4136192-mgmt-1/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4136192-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j4136192-mgmt-1/agentPools/infra1\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4136192-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-93565555-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 3, 2), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4136192-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-93565555-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
11:52:03	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 49b4773 · merge 49b4773	provision	failed	#6585 (merged)	1	provision (1) ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute 11:52:03 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite step graph Show raw failure time=2026-08-24T12:17:22.416Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam" time=2026-08-24T12:17:24.567Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster time=2026-08-24T12:17:26.247Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=b81d2845865e09100d21919fab3e6ce78d71d88f3ac75e34d8fcd2a7b7e34e97 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j7162112-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fb81d2845865e09100d21919fab3e6ce78d71d88f3ac75e34d8fcd2a7b7e34e97 time=2026-08-24T12:25:28.244Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7162112-svc/providers/Microsoft.Resources/deployments/b81d2845865e09100d21919fab3e6ce78d71d88f3ac75e34d8fcd2a7b7e34e97/operationStatuses/08584140330401352534\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7162112-svc/providers/Microsoft.Resources/deployments/cluster-fjywew6dvq3ek\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7162112-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7162112-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7162112-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j7162112-svc/agentPools/u64d8dsv61\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7162112-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-30429984-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 3, 2), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7162112-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-30429984-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
11:48:53	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 2f68bd3	e2e	failed	#6070 (open)	4	e2e (4) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 11:48:53 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260630preview and verify the ingress is internal · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc001600420>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-rhjcsskbzqd7" to finish creating: GET https://rp.j3221632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fdd8d463-3794-4ddc-a4f8-6d580bc2c2a9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fdd8d463-3794-4ddc-a4f8-6d580bc2c2a9", "name": "fdd8d463-3794-4ddc-a4f8-6d580bc2c2a9", "status": "Failed", "startTime": "2026-08-24T12:34:25.475476952Z", "endTime": "2026-08-24T12:53:28.435563679Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc001600420>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-rhjcsskbzqd7" to finish creating: GET https://rp.j3221632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fdd8d463-3794-4ddc-a4f8-6d580bc2c2a9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fdd8d463-3794-4ddc-a4f8-6d580bc2c2a9", "name": "fdd8d463-3794-4ddc-a4f8-6d580bc2c2a9", "status": "Failed", "startTime": "2026-08-24T12:34:25.475476952Z", "endTime": "2026-08-24T12:53:28.435563679Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred UpdateHCPCluster20260630 timed out after <minutes> minutes; context deadline exceeded 11:48:53 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:172]: failed to update cluster with new KMS key Unexpected error: <fmt.wrapErrors \| 0xc00132c390>: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-cflktt657pch" to finish updating, caused by: timeout '30.000000' minutes exceeded during UpdateHCPCluster20260630 for cluster kms-key-rotate-422 in resource group kms-key-rotate-cflktt657pch, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:172]: failed to update cluster with new KMS key Unexpected error: <fmt.wrapErrors \| 0xc00132c390>: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-cflktt657pch" to finish updating, caused by: timeout '30.000000' minutes exceeded during UpdateHCPCluster20260630 for cluster kms-key-rotate-422 in resource group kms-key-rotate-cflktt657pch, error: context deadline exceeded ... occurred expected 3 nodes to have original label key1=value1 11:48:53 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:259]: expected 3 nodes to have original label key1=value1 Expected <bool>: ... to be true fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:259]: expected 3 nodes to have original label key1=value1 Expected <bool>: ... to be true UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded 11:48:53 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:208]: failed to enable autoscaling on node pool np-one-node Unexpected error: <fmt.wrapErrors \| 0xc00396d0e0>: failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-4984p9m7f6lc" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-4984p9m7f6lc, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:208]: failed to enable autoscaling on node pool np-one-node Unexpected error: <fmt.wrapErrors \| 0xc00396d0e0>: failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-4984p9m7f6lc" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-4984p9m7f6lc, error: context deadline exceeded ... occurred
11:34:18	pull-ci-Azure-ARO-HCP-main-e2e-parallel head d1ddfa5	provision	failed	#6424 (open)	1	provision (1) ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute 11:34:18 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite step graph Show raw failure time=2026-08-24T12:10:15.954Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam" time=2026-08-24T12:10:17.764Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster time=2026-08-24T12:10:19.676Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=4fb85e22a773a3391f4c0348e0b5470c084630ac275bea53bc5bf265bbbb61e3 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j0253696-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F4fb85e22a773a3391f4c0348e0b5470c084630ac275bea53bc5bf265bbbb61e3 time=2026-08-24T12:18:21.562Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0253696-svc/providers/Microsoft.Resources/deployments/4fb85e22a773a3391f4c0348e0b5470c084630ac275bea53bc5bf265bbbb61e3/operationStatuses/08584140334670745974\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0253696-svc/providers/Microsoft.Resources/deployments/cluster-tswqsaf7g3tae\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0253696-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0253696-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0253696-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j0253696-svc/agentPools/u64d8dsv61\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0253696-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-37903298-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 3, 2), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0253696-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-37903298-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
11:05:53	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 515e34b	provision	failed	#6583 (open)	1	provision (1) ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute 11:05:53 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite step graph Show raw failure time=2026-08-24T11:34:04.464Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam" time=2026-08-24T11:34:07.135Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster time=2026-08-24T11:34:09.248Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=9b211860af6b0c145b57f6330b1e13702971f10380c81f1e4f30ef18196d35a3 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j0080256-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F9b211860af6b0c145b57f6330b1e13702971f10380c81f1e4f30ef18196d35a3 time=2026-08-24T11:43:11.648Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0080256-svc/providers/Microsoft.Resources/deployments/9b211860af6b0c145b57f6330b1e13702971f10380c81f1e4f30ef18196d35a3/operationStatuses/08584140356374348833\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0080256-svc/providers/Microsoft.Resources/deployments/cluster-dkoaaglt4pes6\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0080256-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0080256-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0080256-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j0080256-svc/agentPools/u64d8dsv61\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0080256-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-10945567-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 3, 2), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0080256-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-10945567-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0080256-svc/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0080256-svc/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0080256-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j0080256-svc/agentPools/infra1\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0080256-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-17123608-vmss'. Please try alternative sizes 'Standard_D8ds_v6' (zones 3, 2), 'Standard_D4ads_v7' (zones 3, 1, 2), 'Standard_D4s_v6' (zones 3, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0080256-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-17123608-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
11:05:38	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 0633420	provision	failed	#6575 (open)	1	provision (1) ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute 11:05:38 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite step graph Show raw failure time=2026-08-24T11:32:20.643Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam" time=2026-08-24T11:32:23.051Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster time=2026-08-24T11:32:24.783Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=74ef5b6bb1b5c6fd3ba7cf88a598fe1556c944274c8643c2c60a526c2eccdb5a portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j5212032-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F74ef5b6bb1b5c6fd3ba7cf88a598fe1556c944274c8643c2c60a526c2eccdb5a time=2026-08-24T11:41:26.957Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5212032-svc/providers/Microsoft.Resources/deployments/74ef5b6bb1b5c6fd3ba7cf88a598fe1556c944274c8643c2c60a526c2eccdb5a/operationStatuses/08584140357416951822\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5212032-svc/providers/Microsoft.Resources/deployments/cluster-3hrc2fvonf4ms\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5212032-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5212032-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5212032-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j5212032-svc/agentPools/u64d8dsv61\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5212032-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-38464807-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 3, 2), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5212032-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-38464807-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
11:05:16	pull-ci-Azure-ARO-HCP-main-e2e-parallel head af0ab1f	provision	failed	#6504 (open)	1	provision (1) ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute 11:05:16 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster (stamp=1) · suite step graph Show raw failure time=2026-08-24T11:29:35.754Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 description="Step cluster\n Kind: ARM\n Template: templates/mgmt-cluster.bicep\n Parameters: configurations/mgmt-cluster.tmpl.bicepparam" time=2026-08-24T11:29:37.122Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 time=2026-08-24T11:29:38.565Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 deployment=06d9da801c490b565266bf3066adb6655831ea121ede6c8b88ef3955cc91c2af portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j9174016-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F06d9da801c490b565266bf3066adb6655831ea121ede6c8b88ef3955cc91c2af time=2026-08-24T11:38:10.769Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9174016-mgmt-1/providers/Microsoft.Resources/deployments/06d9da801c490b565266bf3066adb6655831ea121ede6c8b88ef3955cc91c2af/operationStatuses/08584140359078639543\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9174016-mgmt-1/providers/Microsoft.Resources/deployments/cluster-tztc6ggh322ps\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9174016-mgmt-1/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9174016-mgmt-1/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9174016-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j9174016-mgmt-1/agentPools/infra1\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9174016-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-11156693-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 3, 2), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9174016-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-11156693-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
10:49:11	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 669f42e	provision	failed	#6618 (open)	1	provision (1) ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute 10:49:11 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite step graph Show raw failure time=2026-08-24T10:57:28.271Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam" time=2026-08-24T10:57:30.601Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster time=2026-08-24T10:57:32.308Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=000c9f6cacb3aef7ed0c72330601e9df4ede8a902eb59bd2ba5f467f93427386 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j7975296-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F000c9f6cacb3aef7ed0c72330601e9df4ede8a902eb59bd2ba5f467f93427386 time=2026-08-24T11:12:35.760Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7975296-svc/providers/Microsoft.Resources/deployments/000c9f6cacb3aef7ed0c72330601e9df4ede8a902eb59bd2ba5f467f93427386/operationStatuses/08584140378341181394\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7975296-svc/providers/Microsoft.Resources/deployments/cluster-gxrvf2fjg2hfw\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7975296-svc/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7975296-svc/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7975296-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j7975296-svc/agentPools/infra1\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7975296-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-96441150-vmss'. Please try alternative sizes 'Standard_D8ds_v6' (zones 2, 3), 'Standard_D4ads_v7' (zones 2, 1, 3), 'Standard_D4s_v6' (zones 3, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7975296-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-96441150-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7975296-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7975296-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7975296-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j7975296-svc/agentPools/u64d8dsv61\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7975296-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-57198211-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 3, 2), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7975296-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-57198211-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
10:22:07	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 8d7f552 · merge 8d7f552	n/a	passed	#6656 (merged)	0	n/a
10:00:02	pull-ci-Azure-ARO-HCP-main-e2e-parallel head d33b7bc	provision	failed	#6070 (open)	1	provision (1) ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute 10:00:02 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster (stamp=1) · suite step graph Show raw failure time=2026-08-24T10:24:43.889Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 description="Step cluster\n Kind: ARM\n Template: templates/mgmt-cluster.bicep\n Parameters: configurations/mgmt-cluster.tmpl.bicepparam" time=2026-08-24T10:24:45.005Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 time=2026-08-24T10:24:46.196Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 deployment=b901b6ce5731a24b329db131b89a909d69b5433ca33c57bfdd39a838d9879a8e portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j7726848-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fb901b6ce5731a24b329db131b89a909d69b5433ca33c57bfdd39a838d9879a8e time=2026-08-24T10:33:18.137Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7726848-mgmt-1/providers/Microsoft.Resources/deployments/b901b6ce5731a24b329db131b89a909d69b5433ca33c57bfdd39a838d9879a8e/operationStatuses/08584140397998744039\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7726848-mgmt-1/providers/Microsoft.Resources/deployments/cluster-cnl6ie2kiux2u\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7726848-mgmt-1/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7726848-mgmt-1/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7726848-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j7726848-mgmt-1/agentPools/infra1\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7726848-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-15008703-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 3, 2), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7726848-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-15008703-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
09:57:41	pull-ci-Azure-ARO-HCP-main-e2e-parallel head d31f142	provision	failed	#6429 (open)	1	provision (1) ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute 09:57:41 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite step graph Show raw failure time=2026-08-24T10:07:30.028Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam" time=2026-08-24T10:07:32.346Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster time=2026-08-24T10:07:34.256Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=d1f5df56795df1062698fb6ccb81ccf8210e23e07169776ac2ac1208670e13c1 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j8845824-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fd1f5df56795df1062698fb6ccb81ccf8210e23e07169776ac2ac1208670e13c1 time=2026-08-24T10:17:06.305Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8845824-svc/providers/Microsoft.Resources/deployments/d1f5df56795df1062698fb6ccb81ccf8210e23e07169776ac2ac1208670e13c1/operationStatuses/08584140408323370960\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8845824-svc/providers/Microsoft.Resources/deployments/cluster-ojjgxet7r56au\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8845824-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8845824-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8845824-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j8845824-svc/agentPools/u64d8dsv61\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8845824-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-31090874-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 3, 2), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8845824-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-31090874-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
09:42:18	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 9ec6505	provision	failed	#6433 (open)	1	provision (1) ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute 09:42:18 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite step graph Show raw failure time=2026-08-24T10:09:53.878Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam" time=2026-08-24T10:09:55.657Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster time=2026-08-24T10:09:57.313Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=70eedf005776c816ac9aa5070bf370704152d4348d77e746226f8f6d195d2b70 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2558848-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F70eedf005776c816ac9aa5070bf370704152d4348d77e746226f8f6d195d2b70 time=2026-08-24T10:17:59.005Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2558848-svc/providers/Microsoft.Resources/deployments/70eedf005776c816ac9aa5070bf370704152d4348d77e746226f8f6d195d2b70/operationStatuses/08584140406891133543\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2558848-svc/providers/Microsoft.Resources/deployments/cluster-d25sm5gw5h6au\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2558848-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2558848-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2558848-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j2558848-svc/agentPools/u64d8dsv61\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2558848-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-18815212-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 3, 2), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2558848-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-18815212-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2558848-svc/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2558848-svc/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2558848-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j2558848-svc/agentPools/infra1\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2558848-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-21467642-vmss'. Please try alternative sizes 'Standard_D8ds_v6' (zones 2, 3), 'Standard_D4ads_v7' (zones 2, 1, 3), 'Standard_D4s_v6' (zones 3, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2558848-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-21467642-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
09:14:22	pull-ci-Azure-ARO-HCP-main-e2e-parallel head d31f142	provision	failed	#6429 (open)	1	provision (1) ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute 09:14:22 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster (stamp=2) · suite step graph Show raw failure time=2026-08-24T09:36:25.013Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 description="Step cluster\n Kind: ARM\n Template: templates/mgmt-cluster.bicep\n Parameters: configurations/mgmt-cluster.tmpl.bicepparam" time=2026-08-24T09:36:26.067Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 time=2026-08-24T09:36:27.123Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 deployment=beaeaed83a7cf95a1b09858e4e28288a9e72f993cc66ca81ec00ff1e8e54f826 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j6971264-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fbeaeaed83a7cf95a1b09858e4e28288a9e72f993cc66ca81ec00ff1e8e54f826 time=2026-08-24T09:44:28.907Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6971264-mgmt-2/providers/Microsoft.Resources/deployments/beaeaed83a7cf95a1b09858e4e28288a9e72f993cc66ca81ec00ff1e8e54f826/operationStatuses/08584140426988902505\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6971264-mgmt-2/providers/Microsoft.Resources/deployments/cluster-vig6u5v5k2tlc\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6971264-mgmt-2/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6971264-mgmt-2/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6971264-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j6971264-mgmt-2/agentPools/infra1\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6971264-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-38714764-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 3, 2), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6971264-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-38714764-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
09:05:04	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 0d8584d	provision	failed	#6517 (open)	1	provision (1) ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute 09:05:04 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite step graph Show raw failure time=2026-08-24T09:32:40.860Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam" time=2026-08-24T09:32:42.623Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster time=2026-08-24T09:32:44.680Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=b5c54fde09a66b2790cd440c85cabb05b95df7e9c3a5cfe24051d2a16aa5cb1f portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j0860160-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fb5c54fde09a66b2790cd440c85cabb05b95df7e9c3a5cfe24051d2a16aa5cb1f time=2026-08-24T09:41:16.547Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0860160-svc/providers/Microsoft.Resources/deployments/b5c54fde09a66b2790cd440c85cabb05b95df7e9c3a5cfe24051d2a16aa5cb1f/operationStatuses/08584140429221841580\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0860160-svc/providers/Microsoft.Resources/deployments/cluster-5n4axvaafqqu4\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0860160-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0860160-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0860160-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j0860160-svc/agentPools/u64d8dsv61\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0860160-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-15189735-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 3, 2), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0860160-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-15189735-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
09:00:53	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 0633420	provision	failed	#6575 (open)	1	provision (1) ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute 09:00:53 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite step graph Show raw failure time=2026-08-24T09:24:29.545Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam" time=2026-08-24T09:24:31.294Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster time=2026-08-24T09:24:33.094Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=73468fd8f0c78c5427a4cf7823170085692bbf8052a8fc7aafe06063c9a01e4a portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2432640-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F73468fd8f0c78c5427a4cf7823170085692bbf8052a8fc7aafe06063c9a01e4a time=2026-08-24T09:32:34.962Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2432640-svc/providers/Microsoft.Resources/deployments/73468fd8f0c78c5427a4cf7823170085692bbf8052a8fc7aafe06063c9a01e4a/operationStatuses/08584140434133821616\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2432640-svc/providers/Microsoft.Resources/deployments/cluster-ii7fofvtmntgk\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2432640-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2432640-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2432640-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j2432640-svc/agentPools/u64d8dsv61\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2432640-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-26463169-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 3, 2), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2432640-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-26463169-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2432640-svc/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2432640-svc/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2432640-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j2432640-svc/agentPools/infra1\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2432640-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-17010878-vmss'. Please try alternative sizes 'Standard_D8ds_v6' (zones 2, 3), 'Standard_D4ads_v7' (zones 2, 1, 3), 'Standard_D4s_v6' (zones 3, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2432640-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-17010878-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
08:56:40	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 669f42e	e2e	failed	#6618 (open)	1	e2e (1) DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded 08:56:40 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:131]: failed to verify simple web app runs on v20260901preview cluster "v20260901" Unexpected error: <fmt.wrapError \| 0xc0005da4a0>: DNS for route host agnhost-e2e-sample-app-gr4x8.apps.aro.v20260901.3e97.j6642304.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-gr4x8.apps.aro.v20260901.3e97.j6642304.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-gr4x8.apps.aro.v20260901.3e97.j6642304.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:131]: failed to verify simple web app runs on v20260901preview cluster "v20260901" Unexpected error: <fmt.wrapError \| 0xc0005da4a0>: DNS for route host agnhost-e2e-sample-app-gr4x8.apps.aro.v20260901.3e97.j6642304.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-gr4x8.apps.aro.v20260901.3e97.j6642304.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-gr4x8.apps.aro.v20260901.3e97.j6642304.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded ... occurred
08:50:25	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 3189853	provision	failed	#6655 (open)	1	provision (1) failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded 08:50:25 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster (stamp=1) · suite step graph Show raw failure time=2026-08-24T09:44:43.919Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 description="Step cluster\n Kind: ARM\n Template: templates/mgmt-cluster.bicep\n Parameters: configurations/mgmt-cluster.tmpl.bicepparam" time=2026-08-24T09:44:44.928Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 time=2026-08-24T09:44:46.104Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 deployment=f40d7b7ab0b2d9646dc8b78fbaa252859442c0d4d4549c8765d66346f06ac123 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j4585344-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Ff40d7b7ab0b2d9646dc8b78fbaa252859442c0d4d4549c8765d66346f06ac123 time=2026-08-24T10:14:43.424Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"
08:21:34	pull-ci-Azure-ARO-HCP-main-e2e-parallel head af0ab1f	provision	failed	#6504 (open)	1	provision (1) ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute 08:21:34 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster (stamp=2) · suite step graph Show raw failure time=2026-08-24T08:44:00.342Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 description="Step cluster\n Kind: ARM\n Template: templates/mgmt-cluster.bicep\n Parameters: configurations/mgmt-cluster.tmpl.bicepparam" time=2026-08-24T08:44:01.736Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 time=2026-08-24T08:44:02.811Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 deployment=9d0846032d344f434503069fd459e7a4c951b5611b431c2eb0b0065a78e96029 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j8407424-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F9d0846032d344f434503069fd459e7a4c951b5611b431c2eb0b0065a78e96029 time=2026-08-24T08:52:34.725Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8407424-mgmt-2/providers/Microsoft.Resources/deployments/9d0846032d344f434503069fd459e7a4c951b5611b431c2eb0b0065a78e96029/operationStatuses/08584140458432325611\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8407424-mgmt-2/providers/Microsoft.Resources/deployments/cluster-xw6oxa26fzyyu\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8407424-mgmt-2/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8407424-mgmt-2/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8407424-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j8407424-mgmt-2/agentPools/infra1\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8407424-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-92828001-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 3, 2), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8407424-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-92828001-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
08:14:51	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head af0ab1f	provision	failed	#6504 (open)	1	provision (1) ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute 08:14:51 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster (stamp=1) · suite step graph Show raw failure time=2026-08-24T08:42:01.093Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 description="Step cluster\n Kind: ARM\n Template: templates/mgmt-cluster.bicep\n Parameters: configurations/mgmt-cluster.tmpl.bicepparam" time=2026-08-24T08:42:02.234Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 time=2026-08-24T08:42:03.446Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 deployment=2e3eb598692cc106a7637db2eb5370285b0cbe96999e9aba188997911e08ff10 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1084544-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F2e3eb598692cc106a7637db2eb5370285b0cbe96999e9aba188997911e08ff10 time=2026-08-24T08:50:05.190Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1084544-mgmt-1/providers/Microsoft.Resources/deployments/2e3eb598692cc106a7637db2eb5370285b0cbe96999e9aba188997911e08ff10/operationStatuses/08584140459626748434\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1084544-mgmt-1/providers/Microsoft.Resources/deployments/cluster-squivz75m522c\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1084544-mgmt-1/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1084544-mgmt-1/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1084544-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j1084544-mgmt-1/agentPools/infra1\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1084544-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-33704510-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 3, 2), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1084544-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-33704510-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
07:49:58	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 669f42e	provision	failed	#6618 (open)	1	provision (1) ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute 07:49:58 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite step graph Show raw failure time=2026-08-24T08:12:40.945Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam" time=2026-08-24T08:12:42.670Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster time=2026-08-24T08:12:44.550Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=583e4878952541552a771b0cfcf02356ded24cbe2016bad8aa87f7a845c8c6c7 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j4757760-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F583e4878952541552a771b0cfcf02356ded24cbe2016bad8aa87f7a845c8c6c7 time=2026-08-24T08:21:46.727Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4757760-svc/providers/Microsoft.Resources/deployments/583e4878952541552a771b0cfcf02356ded24cbe2016bad8aa87f7a845c8c6c7/operationStatuses/08584140477221183467\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4757760-svc/providers/Microsoft.Resources/deployments/cluster-ijkmul6obanbq\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4757760-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4757760-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4757760-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j4757760-svc/agentPools/u64d8dsv61\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4757760-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-32457879-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 2, 3), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4757760-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-32457879-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
07:42:01	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 584ff73	provision	failed	⚠#6447 (open)	1	provision (1) ERROR CODE: DeploymentStackDeploymentFailed; provider Microsoft.Compute 07:42:01 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/nodepools (stamp=2) · suite step graph Show raw failure time=2026-08-24T08:15:25.803Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nodepools stamp=2 description="Step nodepools\n Kind: ARMStack\n Template: templates/aks-nodepools.bicep\n Parameters: configurations/mgmt-cluster-nodepools.tmpl.bicepparam" time=2026-08-24T08:18:16.150Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nodepools stamp=2 err="stamp 2: failed to run ARM step: failed to wait for deployment stack at resource group scope: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/locations/westus2/deploymentStackOperationStatus/67d8fd4b-85de-4d57-a829-7ce0bf268c5d\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentStackDeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/locations/westus2/deploymentStackOperationStatus/67d8fd4b-85de-4d57-a829-7ce0bf268c5d\",\n \"name\": \"67d8fd4b-85de-4d57-a829-7ce0bf268c5d\",\n \"status\": \"failed\",\n \"error\": {\n \"code\": \"DeploymentStackDeploymentFailed\",\n \"message\": \"One or more resources could not be deployed. Correlation id: '676c3fe4-b0a1-44f5-8602-c3c05012b379'.\",\n \"details\": [\n {\n \"code\": \"DeploymentFailed\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9428224-mgmt-2/providers/Microsoft.Resources/deployments/l4mv3scto7z6edigthkjdr62jbdrgoxoif33nvakxlwvg-260824084ipd0\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"OverconstrainedZonalAllocationRequest\",\n \"message\": \"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9428224-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-41975975-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 3, 2), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9428224-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-41975975-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n - Availability Zone\\n - Differencing (Ephemeral) Disks\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\n - VM Size\\n\"\n }\n ]\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
07:39:56	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head ad8973f · merge 33f9055	provision	failed	#6278 (merged)	1	provision (1) ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute 07:39:56 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite step graph Show raw failure time=2026-08-24T07:48:49.284Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam" time=2026-08-24T07:48:51.673Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster time=2026-08-24T07:48:53.514Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=c4ffb8eb5fe59fa8f9c215105245e65567b3ae75c58f1894e0f72126f839af18 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j0745216-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fc4ffb8eb5fe59fa8f9c215105245e65567b3ae75c58f1894e0f72126f839af18 time=2026-08-24T07:57:25.627Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0745216-svc/providers/Microsoft.Resources/deployments/c4ffb8eb5fe59fa8f9c215105245e65567b3ae75c58f1894e0f72126f839af18/operationStatuses/08584140491530361585\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0745216-svc/providers/Microsoft.Resources/deployments/cluster-xzssy2rlha6dw\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0745216-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0745216-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0745216-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j0745216-svc/agentPools/u64d8dsv61\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0745216-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-27534110-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 3, 2), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0745216-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-27534110-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
07:05:33	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 0e848b1	n/a	passed	#6519 (open)	0	n/a
06:51:53	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head ad8973f · merge 33f9055	provision	failed	⚠#6278 (merged)	1	provision (1) ERROR CODE: DeploymentFailed; detail code Timeout; detail message Operation is being Canceled due to timeout.; provider Microsoft.EventGrid 06:51:53 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph Show raw failure time=2026-08-24T07:13:06.981Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-08-24T07:13:09.018Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra time=2026-08-24T07:13:10.536Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=9665ebaf931e927bcf7f137293ddd5a9262983f1f2075a0dff43a22749c6dc59 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1860736%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F9665ebaf931e927bcf7f137293ddd5a9262983f1f2075a0dff43a22749c6dc59 time=2026-08-24T07:29:44.382Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1860736/providers/Microsoft.Resources/deployments/9665ebaf931e927bcf7f137293ddd5a9262983f1f2075a0dff43a22749c6dc59/operationStatuses/08584140512958847265\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1860736/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1860736/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j1860736\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"Timeout\\\",\\r\\n \\\"message\\\": \\\"Operation is being Canceled due to timeout.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
06:35:00	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 8d7f552 · merge 8d7f552	provision	failed	#6656 (merged)	1	provision (1) ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute 06:35:00 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite step graph Show raw failure time=2026-08-24T06:57:54.785Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam" time=2026-08-24T06:57:57.820Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster time=2026-08-24T06:57:59.587Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=183371dadaa86003cf4b2e1148d420a38ad735fb1aed96107d051f056b9bc826 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j0609280-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F183371dadaa86003cf4b2e1148d420a38ad735fb1aed96107d051f056b9bc826 time=2026-08-24T07:06:31.480Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0609280-svc/providers/Microsoft.Resources/deployments/183371dadaa86003cf4b2e1148d420a38ad735fb1aed96107d051f056b9bc826/operationStatuses/08584140522069521911\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0609280-svc/providers/Microsoft.Resources/deployments/cluster-xfirgka634tem\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0609280-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0609280-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0609280-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j0609280-svc/agentPools/u64d8dsv61\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0609280-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-45479501-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 3, 2), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0609280-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-45479501-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0609280-svc/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0609280-svc/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0609280-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j0609280-svc/agentPools/infra1\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0609280-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-40810365-vmss'. Please try alternative sizes 'Standard_D8ds_v6' (zones 3, 2), 'Standard_D4ads_v7' (zones 3, 1, 2), 'Standard_D4s_v6' (zones 3, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0609280-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-40810365-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
06:29:54	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 086d692	n/a	passed	#6643 (open)	0	n/a
06:28:37	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 515e34b	provision	failed	#6583 (open)	1	provision (1) ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute 06:28:37 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite step graph Show raw failure time=2026-08-24T07:03:34.745Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam" time=2026-08-24T07:03:36.353Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster time=2026-08-24T07:03:37.884Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=5bdd2a18d3c4d97b3c3ef1c9b862a9a9b55771e7a1021879c9690a8a898ffb61 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j5560320-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F5bdd2a18d3c4d97b3c3ef1c9b862a9a9b55771e7a1021879c9690a8a898ffb61 time=2026-08-24T07:11:39.568Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5560320-svc/providers/Microsoft.Resources/deployments/5bdd2a18d3c4d97b3c3ef1c9b862a9a9b55771e7a1021879c9690a8a898ffb61/operationStatuses/08584140518685563513\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5560320-svc/providers/Microsoft.Resources/deployments/cluster-mssgpgtmbq63s\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5560320-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5560320-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5560320-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j5560320-svc/agentPools/u64d8dsv61\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5560320-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-20641610-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 3, 2), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5560320-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-20641610-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
05:59:42	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 7b9035b	n/a	passed	#6655 (open)	0	n/a
04:53:30	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 6c5cfd0	provision	failed	#6503 (open)	1	provision (1) ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute 04:53:30 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster (stamp=2) · suite step graph Show raw failure time=2026-08-24T05:19:23.165Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 description="Step cluster\n Kind: ARM\n Template: templates/mgmt-cluster.bicep\n Parameters: configurations/mgmt-cluster.tmpl.bicepparam" time=2026-08-24T05:19:24.420Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 time=2026-08-24T05:19:25.641Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 deployment=fbb76f4ae15525e574d930a608886264a04bd969a6a0bd75915b35d608025287 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j0930944-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Ffbb76f4ae15525e574d930a608886264a04bd969a6a0bd75915b35d608025287 time=2026-08-24T05:26:57.916Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0930944-mgmt-2/providers/Microsoft.Resources/deployments/fbb76f4ae15525e574d930a608886264a04bd969a6a0bd75915b35d608025287/operationStatuses/08584140581204682703\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0930944-mgmt-2/providers/Microsoft.Resources/deployments/cluster-wdfog3tlntwm4\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0930944-mgmt-2/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0930944-mgmt-2/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0930944-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j0930944-mgmt-2/agentPools/infra1\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0930944-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-39675041-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 3, 2), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0930944-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-39675041-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
04:39:30	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 515e34b	provision	failed	#6583 (open)	1	provision (1) ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute 04:39:30 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite step graph Show raw failure time=2026-08-24T04:48:17.406Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam" time=2026-08-24T04:48:19.724Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster time=2026-08-24T04:48:21.678Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=2c1fd7cde1ae44aa6b38efdfdd841cda537f1304b159ec745060bb98e153aa01 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2371584-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F2c1fd7cde1ae44aa6b38efdfdd841cda537f1304b159ec745060bb98e153aa01 time=2026-08-24T04:57:23.869Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2371584-svc/providers/Microsoft.Resources/deployments/2c1fd7cde1ae44aa6b38efdfdd841cda537f1304b159ec745060bb98e153aa01/operationStatuses/08584140599851363268\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2371584-svc/providers/Microsoft.Resources/deployments/cluster-6cvwmxyq3qdus\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2371584-svc/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2371584-svc/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2371584-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j2371584-svc/agentPools/infra1\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2371584-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-42381488-vmss'. Please try alternative sizes 'Standard_D8ds_v6' (zones 2, 3), 'Standard_D4ads_v7' (zones 2, 1, 3), 'Standard_D4s_v6' (zones 3, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2371584-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-42381488-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2371584-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2371584-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2371584-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j2371584-svc/agentPools/u64d8dsv61\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2371584-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-11454107-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 3, 2), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2371584-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-11454107-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
04:33:41	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c0a69cc	n/a	passed	#6646 (closed)	0	n/a
03:47:41	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 515e34b	provision	failed	#6583 (open)	1	provision (1) ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute 03:47:41 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite step graph Show raw failure time=2026-08-24T04:17:24.903Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam" time=2026-08-24T04:17:27.266Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster time=2026-08-24T04:17:29.064Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=6967dee589ade651c69bb96507e6cb0c7cb21cecf78950dae52a9a9a4b095938 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j3584256-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F6967dee589ade651c69bb96507e6cb0c7cb21cecf78950dae52a9a9a4b095938 time=2026-08-24T04:26:00.988Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3584256-svc/providers/Microsoft.Resources/deployments/6967dee589ade651c69bb96507e6cb0c7cb21cecf78950dae52a9a9a4b095938/operationStatuses/08584140618375600165\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3584256-svc/providers/Microsoft.Resources/deployments/cluster-skudbwx3rh3io\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3584256-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3584256-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3584256-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j3584256-svc/agentPools/u64d8dsv61\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3584256-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-14718963-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 3, 2), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3584256-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8dsv61-14718963-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
03:45:13	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c0a69cc	provision	failed	#6646 (closed)	1	provision (1) ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute 03:45:13 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster (stamp=1) · suite step graph Show raw failure time=2026-08-24T04:16:06.917Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 description="Step cluster\n Kind: ARM\n Template: templates/mgmt-cluster.bicep\n Parameters: configurations/mgmt-cluster.tmpl.bicepparam" time=2026-08-24T04:16:08.068Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 time=2026-08-24T04:16:10.699Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 deployment=b90b7bfd10ed0c59be2c1babf2e237ad81ed58d2c1a0b3b120a0a0e4de3596a0 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2015744-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fb90b7bfd10ed0c59be2c1babf2e237ad81ed58d2c1a0b3b120a0a0e4de3596a0 time=2026-08-24T04:23:42.372Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2015744-mgmt-1/providers/Microsoft.Resources/deployments/b90b7bfd10ed0c59be2c1babf2e237ad81ed58d2c1a0b3b120a0a0e4de3596a0/operationStatuses/08584140619164865579\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2015744-mgmt-1/providers/Microsoft.Resources/deployments/cluster-mm4ibqsrqe2za\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2015744-mgmt-1/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2015744-mgmt-1/providers/Microsoft.Resources/deployments/infra-agent-pools\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2015744-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j2015744-mgmt-1/agentPools/infra1\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2015744-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-28289844-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 3, 2), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2015744-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-28289844-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n \\\"target\\\": \\\"0\\\",\\r\\n \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n - Availability Zone\\\\n - Differencing (Ephemeral) Disks\\\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n - VM Size\\\\n\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"