CIHealth Run Log

Environment: DEV

Time (UTC)	Job	Failed at	Result	PR	Failed tests	Details
18:14:10	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c403afe	provision	failed	#6638 (open)	1	provision (1) ERROR CODE: DeploymentNotFound 18:14:10 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Fleet.Registration/management/output (stamp=1) · suite step graph Show raw failure time=2026-08-25T18:56:45.688Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=management step=output stamp=1 description="Step output\n Kind: ARM\n Template: ../../dev-infrastructure/templates/output-mgmt.bicep\n Parameters: ../../dev-infrastructure/configurations/output-mgmt.tmpl.bicepparam" time=2026-08-25T18:56:46.086Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=management step=output stamp=1 time=2026-08-25T18:56:46.474Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=management step=output stamp=1 deployment=74b140f8b533b36c1cdb52a5204bd298a1350ffd05ecac7c751c39ee014fab07 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j3363840-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F74b140f8b533b36c1cdb52a5204bd298a1350ffd05ecac7c751c39ee014fab07 time=2026-08-25T18:56:46.527Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=management step=output stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3363840-mgmt-1/providers/Microsoft.Resources/deployments/74b140f8b533b36c1cdb52a5204bd298a1350ffd05ecac7c751c39ee014fab07/operationStatuses/08584139226791753333\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DeploymentNotFound\",\n \"message\": \"Deployment '74b140f8b533b36c1cdb52a5204bd298a1350ffd05ecac7c751c39ee014fab07' could not be found.\"\n }\n}\n--------------------------------------------------------------------------------\n"
18:01:04	pull-ci-Azure-ARO-HCP-main-e2e-parallel head ccc0138	e2e	failed	★#6639 (open)	1	e2e (1) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 18:01:04 UTC · failed at e2e · test SRE can pause schedules to stop backup execution for an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc001022220>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-dm2lv9wtz9pd" to finish creating: GET https://rp.j9302656.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0f240228-e711-46ec-8078-6d29993f3656 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0f240228-e711-46ec-8078-6d29993f3656", "name": "0f240228-e711-46ec-8078-6d29993f3656", "status": "Failed", "startTime": "2026-08-25T18:54:14.698555112Z", "endTime": "2026-08-25T19:13:18.75896335Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc001022220>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-dm2lv9wtz9pd" to finish creating: GET https://rp.j9302656.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0f240228-e711-46ec-8078-6d29993f3656 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0f240228-e711-46ec-8078-6d29993f3656", "name": "0f240228-e711-46ec-8078-6d29993f3656", "status": "Failed", "startTime": "2026-08-25T18:54:14.698555112Z", "endTime": "2026-08-25T19:13:18.75896335Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred
18:00:08	pull-ci-Azure-ARO-HCP-main-e2e-parallel head af0ab1f	e2e	failed	#6504 (open)	5	e2e (5) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 18:00:08 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc000a8e5a0>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-s4m9gpmjnjkn" to finish creating: GET https://rp.j8157056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d8c1ef2-0bb9-4056-b78f-c00010404578 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d8c1ef2-0bb9-4056-b78f-c00010404578", "name": "5d8c1ef2-0bb9-4056-b78f-c00010404578", "status": "Failed", "startTime": "2026-08-25T18:57:32.643817495Z", "endTime": "2026-08-25T19:16:33.085736544Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc000a8e5a0>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-s4m9gpmjnjkn" to finish creating: GET https://rp.j8157056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d8c1ef2-0bb9-4056-b78f-c00010404578 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d8c1ef2-0bb9-4056-b78f-c00010404578", "name": "5d8c1ef2-0bb9-4056-b78f-c00010404578", "status": "Failed", "startTime": "2026-08-25T18:57:32.643817495Z", "endTime": "2026-08-25T19:16:33.085736544Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 18:00:08 UTC · failed at e2e · test FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc001562540>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-t5q77t9nznpj" to finish creating: GET https://rp.j8157056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ba6d65cd-777b-4718-ad6f-f4416ae13858 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ba6d65cd-777b-4718-ad6f-f4416ae13858", "name": "ba6d65cd-777b-4718-ad6f-f4416ae13858", "status": "Failed", "startTime": "2026-08-25T18:57:21.772029384Z", "endTime": "2026-08-25T19:16:23.085277421Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc001562540>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-t5q77t9nznpj" to finish creating: GET https://rp.j8157056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ba6d65cd-777b-4718-ad6f-f4416ae13858 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ba6d65cd-777b-4718-ad6f-f4416ae13858", "name": "ba6d65cd-777b-4718-ad6f-f4416ae13858", "status": "Failed", "startTime": "2026-08-25T18:57:21.772029384Z", "endTime": "2026-08-25T19:16:23.085277421Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 18:00:08 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000dde460>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-4rmtpmn4hkq5" to finish creating: GET https://rp.j8157056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e8c913f3-9cc4-4379-a1e7-07fb33bc22d7 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e8c913f3-9cc4-4379-a1e7-07fb33bc22d7", "name": "e8c913f3-9cc4-4379-a1e7-07fb33bc22d7", "status": "Failed", "startTime": "2026-08-25T18:58:41.575567001Z", "endTime": "2026-08-25T19:17:43.086444554Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000dde460>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-4rmtpmn4hkq5" to finish creating: GET https://rp.j8157056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e8c913f3-9cc4-4379-a1e7-07fb33bc22d7 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e8c913f3-9cc4-4379-a1e7-07fb33bc22d7", "name": "e8c913f3-9cc4-4379-a1e7-07fb33bc22d7", "status": "Failed", "startTime": "2026-08-25T18:58:41.575567001Z", "endTime": "2026-08-25T19:17:43.086444554Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 18:00:08 UTC · failed at e2e · test SRE can pause schedules to stop backup execution for an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc00141cb00>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-mh47z45hngdj" to finish creating: GET https://rp.j8157056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd28938c-3ebd-4b76-94b7-9f92e718f37e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd28938c-3ebd-4b76-94b7-9f92e718f37e", "name": "bd28938c-3ebd-4b76-94b7-9f92e718f37e", "status": "Failed", "startTime": "2026-08-25T18:57:23.514305148Z", "endTime": "2026-08-25T19:16:23.093948104Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc00141cb00>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-mh47z45hngdj" to finish creating: GET https://rp.j8157056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd28938c-3ebd-4b76-94b7-9f92e718f37e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd28938c-3ebd-4b76-94b7-9f92e718f37e", "name": "bd28938c-3ebd-4b76-94b7-9f92e718f37e", "status": "Failed", "startTime": "2026-08-25T18:57:23.514305148Z", "endTime": "2026-08-25T19:16:23.093948104Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 18:00:08 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-pf7hbvppg5pm/cluster-candidate-4-20-42jsf9 should provision Unexpected error: <fmt.wrapError \| 0xc000e192e0>: failed to create HCP cluster cluster-candidate-4-20-42jsf9: failed waiting for cluster="cluster-candidate-4-20-42jsf9" in resourcegroup="rg-candidate-4-20-pf7hbvppg5pm" to finish creating: GET https://rp.j8157056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cbe12fb2-b5e4-45d1-bde4-07c02a203b34 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cbe12fb2-b5e4-45d1-bde4-07c02a203b34", "name": "cbe12fb2-b5e4-45d1-bde4-07c02a203b34", "status": "Failed", "startTime": "2026-08-25T18:57:22.069599707Z", "endTime": "2026-08-25T19:16:23.011359626Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-pf7hbvppg5pm/cluster-candidate-4-20-42jsf9 should provision Unexpected error: <fmt.wrapError \| 0xc000e192e0>: failed to create HCP cluster cluster-candidate-4-20-42jsf9: failed waiting for cluster="cluster-candidate-4-20-42jsf9" in resourcegroup="rg-candidate-4-20-pf7hbvppg5pm" to finish creating: GET https://rp.j8157056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cbe12fb2-b5e4-45d1-bde4-07c02a203b34 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cbe12fb2-b5e4-45d1-bde4-07c02a203b34", "name": "cbe12fb2-b5e4-45d1-bde4-07c02a203b34", "status": "Failed", "startTime": "2026-08-25T18:57:22.069599707Z", "endTime": "2026-08-25T19:16:23.011359626Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred
17:59:09	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 4e67bc7	unknown	failed	#6554 (open)	1	non-artifact-backed failure (no junit artifacts)
17:37:17	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 52e11dc	unknown	failed	#6657 (open)	1	non-artifact-backed failure (no junit artifacts)
17:31:36	pull-ci-Azure-ARO-HCP-main-e2e-parallel head a660b6e	n/a	passed	#6660 (open)	0	n/a
16:56:19	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 4277637	alert, e2e	failed	★#6648 (open)	4	e2e (3) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 16:56:19 UTC · failed at e2e · test FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc00041c4e0>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-6bdpbrgzdzc8" to finish creating: GET https://rp.j9832320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ccc072ba-d2dd-4a86-b29f-6086f60cb482 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ccc072ba-d2dd-4a86-b29f-6086f60cb482", "name": "ccc072ba-d2dd-4a86-b29f-6086f60cb482", "status": "Failed", "startTime": "2026-08-25T17:41:06.665554524Z", "endTime": "2026-08-25T18:00:10.653150122Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc00041c4e0>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-6bdpbrgzdzc8" to finish creating: GET https://rp.j9832320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ccc072ba-d2dd-4a86-b29f-6086f60cb482 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ccc072ba-d2dd-4a86-b29f-6086f60cb482", "name": "ccc072ba-d2dd-4a86-b29f-6086f60cb482", "status": "Failed", "startTime": "2026-08-25T17:41:06.665554524Z", "endTime": "2026-08-25T18:00:10.653150122Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 16:56:19 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-nrmg7s9gn5qf/cluster-candidate-5-0-fh2kkx should provision Unexpected error: <fmt.wrapError \| 0xc0009006a0>: failed to create HCP cluster cluster-candidate-5-0-fh2kkx: failed waiting for cluster="cluster-candidate-5-0-fh2kkx" in resourcegroup="rg-candidate-5-0-nrmg7s9gn5qf" to finish creating: GET https://rp.j9832320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ec762f1c-df9c-4468-b767-2fc6999a5c21 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ec762f1c-df9c-4468-b767-2fc6999a5c21", "name": "ec762f1c-df9c-4468-b767-2fc6999a5c21", "status": "Failed", "startTime": "2026-08-25T17:40:55.279145717Z", "endTime": "2026-08-25T18:00:00.669636178Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-nrmg7s9gn5qf/cluster-candidate-5-0-fh2kkx should provision Unexpected error: <fmt.wrapError \| 0xc0009006a0>: failed to create HCP cluster cluster-candidate-5-0-fh2kkx: failed waiting for cluster="cluster-candidate-5-0-fh2kkx" in resourcegroup="rg-candidate-5-0-nrmg7s9gn5qf" to finish creating: GET https://rp.j9832320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ec762f1c-df9c-4468-b767-2fc6999a5c21 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ec762f1c-df9c-4468-b767-2fc6999a5c21", "name": "ec762f1c-df9c-4468-b767-2fc6999a5c21", "status": "Failed", "startTime": "2026-08-25T17:40:55.279145717Z", "endTime": "2026-08-25T18:00:00.669636178Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 16:56:19 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-t2xtfqljcgg9/cluster-candidate-4-21-g6x6dm should provision Unexpected error: <fmt.wrapError \| 0xc000b8b620>: failed to create HCP cluster cluster-candidate-4-21-g6x6dm: failed waiting for cluster="cluster-candidate-4-21-g6x6dm" in resourcegroup="rg-candidate-4-21-t2xtfqljcgg9" to finish creating: GET https://rp.j9832320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9ea1fb1d-7d0e-4741-b43f-34275080dcae -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9ea1fb1d-7d0e-4741-b43f-34275080dcae", "name": "9ea1fb1d-7d0e-4741-b43f-34275080dcae", "status": "Failed", "startTime": "2026-08-25T17:41:15.401973458Z", "endTime": "2026-08-25T18:00:20.575131507Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-t2xtfqljcgg9/cluster-candidate-4-21-g6x6dm should provision Unexpected error: <fmt.wrapError \| 0xc000b8b620>: failed to create HCP cluster cluster-candidate-4-21-g6x6dm: failed waiting for cluster="cluster-candidate-4-21-g6x6dm" in resourcegroup="rg-candidate-4-21-t2xtfqljcgg9" to finish creating: GET https://rp.j9832320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9ea1fb1d-7d0e-4741-b43f-34275080dcae -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9ea1fb1d-7d0e-4741-b43f-34275080dcae", "name": "9ea1fb1d-7d0e-4741-b43f-34275080dcae", "status": "Failed", "startTime": "2026-08-25T17:41:15.401973458Z", "endTime": "2026-08-25T18:00:20.575131507Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 16:56:19 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 8 time(s) (2 unknown, 6 known) Firing 1: State: Resolved Started: 2026-08-25T17:51:59Z Ended: 2026-08-25T18:18:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9832320-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdob9oikt2alns01e8d0dpcp8roft0r-d6g2f1k7v9j7g7l", pod="router-5d69c54566-gvkv8", severity="warning" Description: Pod ocm-arohcpci01-2sdob9oikt2alns01e8d0dpcp8roft0r-d6g2f1k7v9j7g7l/router-5d69c54566-gvkv8 has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-08-25T17:52:56Z Ended: 2026-08-25T18:15:57Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9832320-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdobf6c1r1ht9a0tq9k4goo675i5316-m4g9c3h1h9r9b2n", pod="router-d7b75fd4b-bsdjf", severity="warning" Description: Pod ocm-arohcpci01-2sdobf6c1r1ht9a0tq9k4goo675i5316-m4g9c3h1h9r9b2n/router-d7b75fd4b-bsdjf has been in a non-ready state for longer than 5 minutes.
16:12:48	pull-ci-Azure-ARO-HCP-main-e2e-parallel head ea2dde0	alert, e2e	failed	#5958 (open)	3	e2e (2) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 16:12:48 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-sgkx2hv5dw5t/cluster-candidate-4-21-jn7kfp should provision Unexpected error: <fmt.wrapError \| 0xc001b1c3a0>: failed to create HCP cluster cluster-candidate-4-21-jn7kfp: failed waiting for cluster="cluster-candidate-4-21-jn7kfp" in resourcegroup="rg-candidate-4-21-sgkx2hv5dw5t" to finish creating: GET https://rp.j9793920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ab3cc2a4-9ba0-4171-85d4-cdbb942a29a4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ab3cc2a4-9ba0-4171-85d4-cdbb942a29a4", "name": "ab3cc2a4-9ba0-4171-85d4-cdbb942a29a4", "status": "Failed", "startTime": "2026-08-25T16:54:12.093832515Z", "endTime": "2026-08-25T17:13:15.538537349Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-sgkx2hv5dw5t/cluster-candidate-4-21-jn7kfp should provision Unexpected error: <fmt.wrapError \| 0xc001b1c3a0>: failed to create HCP cluster cluster-candidate-4-21-jn7kfp: failed waiting for cluster="cluster-candidate-4-21-jn7kfp" in resourcegroup="rg-candidate-4-21-sgkx2hv5dw5t" to finish creating: GET https://rp.j9793920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ab3cc2a4-9ba0-4171-85d4-cdbb942a29a4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ab3cc2a4-9ba0-4171-85d4-cdbb942a29a4", "name": "ab3cc2a4-9ba0-4171-85d4-cdbb942a29a4", "status": "Failed", "startTime": "2026-08-25T16:54:12.093832515Z", "endTime": "2026-08-25T17:13:15.538537349Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 16:12:48 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--tfndrjk9sqsf/cluster-hs-pre-xzndj9 should provision Unexpected error: <fmt.wrapError \| 0xc0010886c0>: failed to create HCP cluster cluster-hs-pre-xzndj9: failed waiting for cluster="cluster-hs-pre-xzndj9" in resourcegroup="rg-hs-pre--tfndrjk9sqsf" to finish creating: GET https://rp.j9793920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1822fb23-da51-4cef-a546-5dab91c6a66e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1822fb23-da51-4cef-a546-5dab91c6a66e", "name": "1822fb23-da51-4cef-a546-5dab91c6a66e", "status": "Failed", "startTime": "2026-08-25T16:54:12.394756997Z", "endTime": "2026-08-25T17:13:15.586924639Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--tfndrjk9sqsf/cluster-hs-pre-xzndj9 should provision Unexpected error: <fmt.wrapError \| 0xc0010886c0>: failed to create HCP cluster cluster-hs-pre-xzndj9: failed waiting for cluster="cluster-hs-pre-xzndj9" in resourcegroup="rg-hs-pre--tfndrjk9sqsf" to finish creating: GET https://rp.j9793920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1822fb23-da51-4cef-a546-5dab91c6a66e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1822fb23-da51-4cef-a546-5dab91c6a66e", "name": "1822fb23-da51-4cef-a546-5dab91c6a66e", "status": "Failed", "startTime": "2026-08-25T16:54:12.394756997Z", "endTime": "2026-08-25T17:13:15.586924639Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 16:12:48 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-25T17:06:23Z Ended: 2026-08-25T17:27:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9793920-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdnld7nfnceg8m66m97qmk82pp6t0ti-v9d8m6w5n0e5p1k", pod="router-79c9f75898-m4z8p", severity="warning" Description: Pod ocm-arohcpci01-2sdnld7nfnceg8m66m97qmk82pp6t0ti-v9d8m6w5n0e5p1k/router-79c9f75898-m4z8p has been in a non-ready state for longer than 5 minutes.
16:12:43	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 7eff23c	alert, e2e	failed	#6575 (open)	3	e2e (2) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 16:12:43 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc001132120>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-99sdw8x8fbq7" to finish creating: GET https://rp.j8937216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/577acfc9-e788-457f-b3ee-99ff00da118b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/577acfc9-e788-457f-b3ee-99ff00da118b", "name": "577acfc9-e788-457f-b3ee-99ff00da118b", "status": "Failed", "startTime": "2026-08-25T16:59:34.564176682Z", "endTime": "2026-08-25T17:18:37.881780961Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc001132120>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-99sdw8x8fbq7" to finish creating: GET https://rp.j8937216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/577acfc9-e788-457f-b3ee-99ff00da118b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/577acfc9-e788-457f-b3ee-99ff00da118b", "name": "577acfc9-e788-457f-b3ee-99ff00da118b", "status": "Failed", "startTime": "2026-08-25T16:59:34.564176682Z", "endTime": "2026-08-25T17:18:37.881780961Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 16:12:43 UTC · failed at e2e · test FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc000830340>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-4d62fjrxvwg7" to finish creating: GET https://rp.j8937216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d1eaea7a-aae6-4966-8ffc-c5e041d39c1c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d1eaea7a-aae6-4966-8ffc-c5e041d39c1c", "name": "d1eaea7a-aae6-4966-8ffc-c5e041d39c1c", "status": "Failed", "startTime": "2026-08-25T16:59:44.084689304Z", "endTime": "2026-08-25T17:18:47.865081649Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc000830340>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-4d62fjrxvwg7" to finish creating: GET https://rp.j8937216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d1eaea7a-aae6-4966-8ffc-c5e041d39c1c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d1eaea7a-aae6-4966-8ffc-c5e041d39c1c", "name": "d1eaea7a-aae6-4966-8ffc-c5e041d39c1c", "status": "Failed", "startTime": "2026-08-25T16:59:44.084689304Z", "endTime": "2026-08-25T17:18:47.865081649Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 16:12:43 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-25T17:12:13Z Ended: 2026-08-25T17:32:13Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8937216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdnntqml99r0h8mnakh8p60me0um74a-f3v4t8c7c6p2p7s", pod="router-6b855cc9c6-fxfpr", severity="warning" Description: Pod ocm-arohcpci01-2sdnntqml99r0h8mnakh8p60me0um74a-f3v4t8c7c6p2p7s/router-6b855cc9c6-fxfpr has been in a non-ready state for longer than 5 minutes.
16:12:42	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c9b8452	alert	failed	#6584 (open)	1	alert (1) alert [hcp] KubePodNotReady fired 16:12:42 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) (1 unknown, 1 known) Firing 1: State: Resolved Started: 2026-08-25T17:21:22Z Ended: 2026-08-25T17:27:16Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7902848-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdns5ree69mkiips5b97sq4nvu4khh3-y9s8x4j8f0u0h4h", pod="router-7857668d8b-4lm5j", severity="warning" Description: Pod ocm-arohcpci01-2sdns5ree69mkiips5b97sq4nvu4khh3-y9s8x4j8f0u0h4h/router-7857668d8b-4lm5j has been in a non-ready state for longer than 5 minutes.
16:10:00	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 53966db	e2e	failed	#6630 (open)	1	e2e (1) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 16:10:00 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc00100b1a0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-frm9xtk8fnxw" to finish creating: GET https://rp.j1790336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8fcd9358-9c18-42ba-8b4c-182393bd49da -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8fcd9358-9c18-42ba-8b4c-182393bd49da", "name": "8fcd9358-9c18-42ba-8b4c-182393bd49da", "status": "Failed", "startTime": "2026-08-25T17:07:37.992364096Z", "endTime": "2026-08-25T17:26:38.742090983Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc00100b1a0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-frm9xtk8fnxw" to finish creating: GET https://rp.j1790336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8fcd9358-9c18-42ba-8b4c-182393bd49da -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8fcd9358-9c18-42ba-8b4c-182393bd49da", "name": "8fcd9358-9c18-42ba-8b4c-182393bd49da", "status": "Failed", "startTime": "2026-08-25T17:07:37.992364096Z", "endTime": "2026-08-25T17:26:38.742090983Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred
15:58:02	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 3e7f850	n/a	passed	#6232 (open)	0	n/a
15:55:04	pull-ci-Azure-ARO-HCP-main-e2e-parallel head e33b5d3	alert, e2e	failed	⚠#6679 (open)	11	e2e (9) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 15:55:04 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000f9eb40>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-nmp69vm78h4k" to finish creating: GET https://rp.j0166528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/584d4084-30a5-4233-a069-f2a0cf2b9043 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/584d4084-30a5-4233-a069-f2a0cf2b9043", "name": "584d4084-30a5-4233-a069-f2a0cf2b9043", "status": "Failed", "startTime": "2026-08-25T16:44:03.375670894Z", "endTime": "2026-08-25T17:03:09.343723737Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000f9eb40>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-nmp69vm78h4k" to finish creating: GET https://rp.j0166528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/584d4084-30a5-4233-a069-f2a0cf2b9043 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/584d4084-30a5-4233-a069-f2a0cf2b9043", "name": "584d4084-30a5-4233-a069-f2a0cf2b9043", "status": "Failed", "startTime": "2026-08-25T16:44:03.375670894Z", "endTime": "2026-08-25T17:03:09.343723737Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred cluster entered terminal Failed state after role assignment deployment 15:55:04 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.000s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.000s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 15:55:04 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc000e502a0>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-nqflzjdjzf4p" to finish creating: GET https://rp.j0166528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/293580a0-972a-4849-9fbf-bbd6db570984 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/293580a0-972a-4849-9fbf-bbd6db570984", "name": "293580a0-972a-4849-9fbf-bbd6db570984", "status": "Failed", "startTime": "2026-08-25T16:42:29.605638715Z", "endTime": "2026-08-25T17:01:29.31272361Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc000e502a0>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-nqflzjdjzf4p" to finish creating: GET https://rp.j0166528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/293580a0-972a-4849-9fbf-bbd6db570984 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/293580a0-972a-4849-9fbf-bbd6db570984", "name": "293580a0-972a-4849-9fbf-bbd6db570984", "status": "Failed", "startTime": "2026-08-25T16:42:29.605638715Z", "endTime": "2026-08-25T17:01:29.31272361Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 15:55:04 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-xv5r4nm664qq/cluster-candidate-5-0-t7khkn should provision Unexpected error: <fmt.wrapError \| 0xc001240400>: failed to create HCP cluster cluster-candidate-5-0-t7khkn: failed waiting for cluster="cluster-candidate-5-0-t7khkn" in resourcegroup="rg-candidate-5-0-xv5r4nm664qq" to finish creating: GET https://rp.j0166528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b985677c-5f0e-4eaf-9750-2cdf3f5d20fd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b985677c-5f0e-4eaf-9750-2cdf3f5d20fd", "name": "b985677c-5f0e-4eaf-9750-2cdf3f5d20fd", "status": "Failed", "startTime": "2026-08-25T16:42:32.948401484Z", "endTime": "2026-08-25T17:01:39.319371765Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-xv5r4nm664qq/cluster-candidate-5-0-t7khkn should provision Unexpected error: <fmt.wrapError \| 0xc001240400>: failed to create HCP cluster cluster-candidate-5-0-t7khkn: failed waiting for cluster="cluster-candidate-5-0-t7khkn" in resourcegroup="rg-candidate-5-0-xv5r4nm664qq" to finish creating: GET https://rp.j0166528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b985677c-5f0e-4eaf-9750-2cdf3f5d20fd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b985677c-5f0e-4eaf-9750-2cdf3f5d20fd", "name": "b985677c-5f0e-4eaf-9750-2cdf3f5d20fd", "status": "Failed", "startTime": "2026-08-25T16:42:32.948401484Z", "endTime": "2026-08-25T17:01:39.319371765Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 15:55:04 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc000881000>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-lwkzhddxfk2r" to finish creating: GET https://rp.j0166528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8d98a06a-da1b-47f1-8998-4d29a4c0c76a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8d98a06a-da1b-47f1-8998-4d29a4c0c76a", "name": "8d98a06a-da1b-47f1-8998-4d29a4c0c76a", "status": "Failed", "startTime": "2026-08-25T16:44:47.284835124Z", "endTime": "2026-08-25T17:03:49.355549371Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc000881000>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-lwkzhddxfk2r" to finish creating: GET https://rp.j0166528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8d98a06a-da1b-47f1-8998-4d29a4c0c76a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8d98a06a-da1b-47f1-8998-4d29a4c0c76a", "name": "8d98a06a-da1b-47f1-8998-4d29a4c0c76a", "status": "Failed", "startTime": "2026-08-25T16:44:47.284835124Z", "endTime": "2026-08-25T17:03:49.355549371Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator, router]; provider Microsoft.RedHatOpenShift 15:55:04 UTC · failed at e2e · test SRE can pause schedules to stop backup execution for an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000ea0340>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-cm65c9f47lvs" to finish creating: GET https://rp.j0166528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/87f7df17-c6af-4eeb-b82d-34b9b0fdeac2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/87f7df17-c6af-4eeb-b82d-34b9b0fdeac2", "name": "87f7df17-c6af-4eeb-b82d-34b9b0fdeac2", "status": "Failed", "startTime": "2026-08-25T16:42:45.179574291Z", "endTime": "2026-08-25T17:01:49.211726949Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000ea0340>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-cm65c9f47lvs" to finish creating: GET https://rp.j0166528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/87f7df17-c6af-4eeb-b82d-34b9b0fdeac2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/87f7df17-c6af-4eeb-b82d-34b9b0fdeac2", "name": "87f7df17-c6af-4eeb-b82d-34b9b0fdeac2", "status": "Failed", "startTime": "2026-08-25T16:42:45.179574291Z", "endTime": "2026-08-25T17:01:49.211726949Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 15:55:04 UTC · failed at e2e · test FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc000a542e0>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-wdvkthrhsbcl" to finish creating: GET https://rp.j0166528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a7aecbab-9fd1-473f-9e9a-03ff3a146f3e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a7aecbab-9fd1-473f-9e9a-03ff3a146f3e", "name": "a7aecbab-9fd1-473f-9e9a-03ff3a146f3e", "status": "Failed", "startTime": "2026-08-25T16:42:33.26817262Z", "endTime": "2026-08-25T17:01:39.343970634Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc000a542e0>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-wdvkthrhsbcl" to finish creating: GET https://rp.j0166528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a7aecbab-9fd1-473f-9e9a-03ff3a146f3e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a7aecbab-9fd1-473f-9e9a-03ff3a146f3e", "name": "a7aecbab-9fd1-473f-9e9a-03ff3a146f3e", "status": "Failed", "startTime": "2026-08-25T16:42:33.26817262Z", "endTime": "2026-08-25T17:01:39.343970634Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade) 15:55:04 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:220]: Timed out after 2700.000s. Expected success, but got an error: <errors.joinError \| 0xc000012048>: VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade) VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.22, targetMinor=5.0) failed: clusterversion status.history has no version in target minor "5.0" ... fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:220]: Timed out after 2700.000s. Expected success, but got an error: <errors.joinError \| 0xc000012048>: VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade) VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.22, targetMinor=5.0) failed: clusterversion status.history has no version in target minor "5.0" ... ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 15:55:04 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc00052b120>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-4bcbzx62gx2r" to finish creating: GET https://rp.j0166528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/25c719fd-fce5-488f-a4c7-766d0744c777 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/25c719fd-fce5-488f-a4c7-766d0744c777", "name": "25c719fd-fce5-488f-a4c7-766d0744c777", "status": "Failed", "startTime": "2026-08-25T16:44:04.332076109Z", "endTime": "2026-08-25T17:03:09.229572599Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc00052b120>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-4bcbzx62gx2r" to finish creating: GET https://rp.j0166528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/25c719fd-fce5-488f-a4c7-766d0744c777 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/25c719fd-fce5-488f-a4c7-766d0744c777", "name": "25c719fd-fce5-488f-a4c7-766d0744c777", "status": "Failed", "startTime": "2026-08-25T16:44:04.332076109Z", "endTime": "2026-08-25T17:03:09.229572599Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (2) alert [hcp] KubePodNotReady fired 15:55:04 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-25T16:56:51Z Ended: 2026-08-25T17:13:55Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0166528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdnfuacmv6agt0dlstsphj72ajf634i-t5v3e1s2j1a3d2t", pod="router-77cd575f5f-nmfnc", severity="warning" Description: Pod ocm-arohcpci01-2sdnfuacmv6agt0dlstsphj72ajf634i-t5v3e1s2j1a3d2t/router-77cd575f5f-nmfnc has been in a non-ready state for longer than 5 minutes. alert [svc] BackendControllerRetryHotLoop fired 15:55:04 UTC · failed at alert · test [aro-hcp-observability] [svc] alert BackendControllerRetryHotLoop does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-08-25T17:40:35Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j0166528-svc", component="backend", name="triggercontrolplaneupgrade", severity="warning" Description: Backend controller workqueue triggercontrolplaneupgrade has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work.
15:23:59	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c1274d7	alert, e2e	failed	#6070 (open)	9	e2e (7) UpdateHCPCluster20260630 timed out after <minutes> minutes; context deadline exceeded 15:23:59 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:172]: failed to update cluster with new KMS key Unexpected error: <fmt.wrapErrors \| 0xc00110af90>: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-72g5866h88vj" to finish updating, caused by: timeout '30.000000' minutes exceeded during UpdateHCPCluster20260630 for cluster kms-key-rotate-422 in resource group kms-key-rotate-72g5866h88vj, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:172]: failed to update cluster with new KMS key Unexpected error: <fmt.wrapErrors \| 0xc00110af90>: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-72g5866h88vj" to finish updating, caused by: timeout '30.000000' minutes exceeded during UpdateHCPCluster20260630 for cluster kms-key-rotate-422 in resource group kms-key-rotate-72g5866h88vj, error: context deadline exceeded ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 15:23:59 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc000e40258>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-m7ds2xdkbpmr" to finish deleting: GET https://rp.j0912128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/424a05dd-5f96-45bf-a866-906148cef205 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/424a05dd-5f96-45bf-a866-906148cef205", "name": "424a05dd-5f96-45bf-a866-906148cef205", "status": "Failed", "startTime": "2026-08-25T16:34:49.049807041Z", "endTime": "2026-08-25T16:58:58.538368578Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sdn6lgco9mj0rdvfe15crftvs8408gm still exists (deletion dispatched at 2026-08-25T16:35:30Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc000e40258>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-m7ds2xdkbpmr" to finish deleting: GET https://rp.j0912128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/424a05dd-5f96-45bf-a866-906148cef205 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/424a05dd-5f96-45bf-a866-906148cef205", "name": "424a05dd-5f96-45bf-a866-906148cef205", "status": "Failed", "startTime": "2026-08-25T16:34:49.049807041Z", "endTime": "2026-08-25T16:58:58.538368578Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sdn6lgco9mj0rdvfe15crftvs8408gm still exists (deletion dispatched at 2026-08-25T16:35:30Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred expected 3 nodes to have original label key1=value1 15:23:59 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:259]: expected 3 nodes to have original label key1=value1 Expected <bool>: ... to be true fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:259]: expected 3 nodes to have original label key1=value1 Expected <bool>: ... to be true UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded 15:23:59 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:208]: failed to enable autoscaling on node pool np-one-node Unexpected error: <fmt.wrapErrors \| 0xc0033fc180>: failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-jz5t9hfw7w26" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-jz5t9hfw7w26, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:208]: failed to enable autoscaling on node pool np-one-node Unexpected error: <fmt.wrapErrors \| 0xc0033fc180>: failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-jz5t9hfw7w26" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-jz5t9hfw7w26, error: context deadline exceeded ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 15:23:59 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc001a36438>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-vkm2hsvvkr8h" to finish deleting: GET https://rp.j0912128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ba3767be-4721-4bca-bfeb-3c6890864f09 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ba3767be-4721-4bca-bfeb-3c6890864f09", "name": "ba3767be-4721-4bca-bfeb-3c6890864f09", "status": "Failed", "startTime": "2026-08-25T16:44:24.055129576Z", "endTime": "2026-08-25T17:08:28.536915865Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sdn6vvfklofaj57683jl810auik73me still exists (deletion dispatched at 2026-08-25T16:44:25Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc001a36438>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-vkm2hsvvkr8h" to finish deleting: GET https://rp.j0912128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ba3767be-4721-4bca-bfeb-3c6890864f09 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ba3767be-4721-4bca-bfeb-3c6890864f09", "name": "ba3767be-4721-4bca-bfeb-3c6890864f09", "status": "Failed", "startTime": "2026-08-25T16:44:24.055129576Z", "endTime": "2026-08-25T17:08:28.536915865Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sdn6vvfklofaj57683jl810auik73me still exists (deletion dispatched at 2026-08-25T16:44:25Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 15:23:59 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc0019ac228>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-lhs7stnz7jdx" to finish deleting: GET https://rp.j0912128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ee794509-c973-4d3d-9aac-193a29f17172 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ee794509-c973-4d3d-9aac-193a29f17172", "name": "ee794509-c973-4d3d-9aac-193a29f17172", "status": "Failed", "startTime": "2026-08-25T16:40:54.657536996Z", "endTime": "2026-08-25T17:04:58.539200774Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sdn70pm4stdfpd62ob77dhptdq67h82 still exists (deletion dispatched at 2026-08-25T16:40:56Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc0019ac228>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-lhs7stnz7jdx" to finish deleting: GET https://rp.j0912128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ee794509-c973-4d3d-9aac-193a29f17172 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ee794509-c973-4d3d-9aac-193a29f17172", "name": "ee794509-c973-4d3d-9aac-193a29f17172", "status": "Failed", "startTime": "2026-08-25T16:40:54.657536996Z", "endTime": "2026-08-25T17:04:58.539200774Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sdn70pm4stdfpd62ob77dhptdq67h82 still exists (deletion dispatched at 2026-08-25T16:40:56Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 15:23:59 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc001a3a078>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-vwd6cf59jxq2" to finish deleting: GET https://rp.j0912128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8cb828c3-349f-4ff7-bb15-1cb33bc05bfa -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8cb828c3-349f-4ff7-bb15-1cb33bc05bfa", "name": "8cb828c3-349f-4ff7-bb15-1cb33bc05bfa", "status": "Failed", "startTime": "2026-08-25T16:34:33.14717886Z", "endTime": "2026-08-25T16:58:38.634142205Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sdn6lk5i59n2vdugq91i3qhl1hovg7u still exists (deletion dispatched at 2026-08-25T16:34:39Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc001a3a078>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-vwd6cf59jxq2" to finish deleting: GET https://rp.j0912128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8cb828c3-349f-4ff7-bb15-1cb33bc05bfa -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8cb828c3-349f-4ff7-bb15-1cb33bc05bfa", "name": "8cb828c3-349f-4ff7-bb15-1cb33bc05bfa", "status": "Failed", "startTime": "2026-08-25T16:34:33.14717886Z", "endTime": "2026-08-25T16:58:38.634142205Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sdn6lk5i59n2vdugq91i3qhl1hovg7u still exists (deletion dispatched at 2026-08-25T16:34:39Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (2) alert [svc] BackendControllerRetryHotLoop fired 15:23:59 UTC · failed at alert · test [aro-hcp-observability] [svc] alert BackendControllerRetryHotLoop does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-25T16:39:54Z Ended: 2026-08-25T17:06:55Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j0912128-svc", component="backend", name="clusterresourcescontroller", severity="warning" Description: Backend controller workqueue clusterresourcescontroller has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. alert [hcp] KubeDeploymentReplicasMismatch fired 15:23:59 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire · suite aro-hcp-tests Show raw failure alert fired 14 time(s) Firing 1: State: Fired (not resolved) Started: 2026-08-25T17:23:07Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0912128-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.142:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0912128/providers/microsoft.monitor/accounts/hcps-j0912128", namespace="ocm-arohcpci01-2sdn70pm4stdfpd62ob77dhptdq67h82-k7b0n1g7e9t6i9r", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-d2656", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2sdn70pm4stdfpd62ob77dhptdq67h82-k7b0n1g7e9t6i9r/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Fired (not resolved) Started: 2026-08-25T17:23:07Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0912128-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.142:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0912128/providers/microsoft.monitor/accounts/hcps-j0912128", namespace="ocm-arohcpci01-2sdn70pm4stdfpd62ob77dhptdq67h82-k7b0n1g7e9t6i9r", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-d2656", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2sdn70pm4stdfpd62ob77dhptdq67h82-k7b0n1g7e9t6i9r/cluster-api has not matched the expected number of replicas for longer than 30 minutes. Firing 3: State: Fired (not resolved) Started: 2026-08-25T17:29:04Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0912128-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.136:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0912128/providers/microsoft.monitor/accounts/hcps-j0912128", namespace="ocm-arohcpci01-2sdn6lk5i59n2vdugq91i3qhl1hovg7u-basic-cluster", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-jcll8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2sdn6lk5i59n2vdugq91i3qhl1hovg7u-basic-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 4: State: Fired (not resolved) Started: 2026-08-25T17:29:04Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0912128-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.136:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0912128/providers/microsoft.monitor/accounts/hcps-j0912128", namespace="ocm-arohcpci01-2sdn6lk5i59n2vdugq91i3qhl1hovg7u-basic-cluster", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-jcll8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2sdn6lk5i59n2vdugq91i3qhl1hovg7u-basic-cluster/cluster-api has not matched the expected number of replicas for longer than 30 minutes. Firing 5: State: Fired (not resolved) Started: 2026-08-25T17:29:04Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0912128-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.136:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0912128/providers/microsoft.monitor/accounts/hcps-j0912128", namespace="ocm-arohcpci01-2sdn6lgco9mj0rdvfe15crftvs8408gm-i3f3a1k1t0i4d3n", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-jcll8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2sdn6lgco9mj0rdvfe15crftvs8408gm-i3f3a1k1t0i4d3n/cluster-api has not matched the expected number of replicas for longer than 30 minutes. Firing 6: State: Fired (not resolved) Started: 2026-08-25T17:29:04Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0912128-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.136:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0912128/providers/microsoft.monitor/accounts/hcps-j0912128", namespace="ocm-arohcpci01-2sdn6vvfklofaj57683jl810auik73me-u1n3j6u1c5a9g2d", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-jcll8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2sdn6vvfklofaj57683jl810auik73me-u1n3j6u1c5a9g2d/cluster-api has not matched the expected number of replicas for longer than 30 minutes. Firing 7: State: Fired (not resolved) Started: 2026-08-25T17:29:04Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0912128-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.136:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0912128/providers/microsoft.monitor/accounts/hcps-j0912128", namespace="ocm-arohcpci01-2sdn6lgco9mj0rdvfe15crftvs8408gm-i3f3a1k1t0i4d3n", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-jcll8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2sdn6lgco9mj0rdvfe15crftvs8408gm-i3f3a1k1t0i4d3n/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 8: State: Fired (not resolved) Started: 2026-08-25T17:29:04Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0912128-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.136:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0912128/providers/microsoft.monitor/accounts/hcps-j0912128", namespace="ocm-arohcpci01-2sdn6vvfklofaj57683jl810auik73me-u1n3j6u1c5a9g2d", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-jcll8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2sdn6vvfklofaj57683jl810auik73me-u1n3j6u1c5a9g2d/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 9: State: Fired (not resolved) Started: 2026-08-25T17:30:05Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0912128-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.136:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0912128/providers/microsoft.monitor/accounts/hcps-j0912128", namespace="ocm-arohcpci01-2sdn6lgco9mj0rdvfe15crftvs8408gm-i3f3a1k1t0i4d3n", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-jcll8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2sdn6lgco9mj0rdvfe15crftvs8408gm-i3f3a1k1t0i4d3n/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 10: State: Fired (not resolved) Started: 2026-08-25T17:30:05Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0912128-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.136:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0912128/providers/microsoft.monitor/accounts/hcps-j0912128", namespace="ocm-arohcpci01-2sdn6vvfklofaj57683jl810auik73me-u1n3j6u1c5a9g2d", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-jcll8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2sdn6vvfklofaj57683jl810auik73me-u1n3j6u1c5a9g2d/cluster-api has not matched the expected number of replicas for longer than 30 minutes. Firing 11: State: Fired (not resolved) Started: 2026-08-25T17:30:05Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0912128-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.136:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0912128/providers/microsoft.monitor/accounts/hcps-j0912128", namespace="ocm-arohcpci01-2sdn6lk5i59n2vdugq91i3qhl1hovg7u-basic-cluster", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-jcll8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2sdn6lk5i59n2vdugq91i3qhl1hovg7u-basic-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 12: State: Fired (not resolved) Started: 2026-08-25T17:30:05Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0912128-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.136:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0912128/providers/microsoft.monitor/accounts/hcps-j0912128", namespace="ocm-arohcpci01-2sdn6lk5i59n2vdugq91i3qhl1hovg7u-basic-cluster", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-jcll8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2sdn6lk5i59n2vdugq91i3qhl1hovg7u-basic-cluster/cluster-api has not matched the expected number of replicas for longer than 30 minutes. Firing 13: State: Fired (not resolved) Started: 2026-08-25T17:30:05Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0912128-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.136:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0912128/providers/microsoft.monitor/accounts/hcps-j0912128", namespace="ocm-arohcpci01-2sdn6lgco9mj0rdvfe15crftvs8408gm-i3f3a1k1t0i4d3n", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-jcll8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2sdn6lgco9mj0rdvfe15crftvs8408gm-i3f3a1k1t0i4d3n/cluster-api has not matched the expected number of replicas for longer than 30 minutes. Firing 14: State: Fired (not resolved) Started: 2026-08-25T17:30:05Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0912128-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.136:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0912128/providers/microsoft.monitor/accounts/hcps-j0912128", namespace="ocm-arohcpci01-2sdn6vvfklofaj57683jl810auik73me-u1n3j6u1c5a9g2d", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-jcll8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2sdn6vvfklofaj57683jl810auik73me-u1n3j6u1c5a9g2d/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
15:17:36	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 46e74dc	n/a	passed	#6680 (open)	0	n/a
15:16:27	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 9749b2d	e2e	failed	⚠#4566 (open)	5	e2e (5) ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [openshift-apiserver]; provider Microsoft.RedHatOpenShift 15:16:27 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc001350960>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-44brmcrqlwhd" to finish creating: GET https://rp.j0599680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bc250653-c8e3-42dd-bdb2-80628cccd921 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bc250653-c8e3-42dd-bdb2-80628cccd921", "name": "bc250653-c8e3-42dd-bdb2-80628cccd921", "status": "Failed", "startTime": "2026-08-25T16:24:55.65138247Z", "endTime": "2026-08-25T16:43:58.821172119Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: community-operators-catalog, redhat-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: openshift-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc001350960>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-44brmcrqlwhd" to finish creating: GET https://rp.j0599680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bc250653-c8e3-42dd-bdb2-80628cccd921 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bc250653-c8e3-42dd-bdb2-80628cccd921", "name": "bc250653-c8e3-42dd-bdb2-80628cccd921", "status": "Failed", "startTime": "2026-08-25T16:24:55.65138247Z", "endTime": "2026-08-25T16:43:58.821172119Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: community-operators-catalog, redhat-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: openshift-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 15:16:27 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-xhfxlf4rcz52/cluster-candidate-4-21-rvhxrv should provision Unexpected error: <fmt.wrapError \| 0xc000f54d40>: failed to create HCP cluster cluster-candidate-4-21-rvhxrv: failed waiting for cluster="cluster-candidate-4-21-rvhxrv" in resourcegroup="rg-candidate-4-21-xhfxlf4rcz52" to finish creating: GET https://rp.j0599680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/62063914-1f6c-4e07-b0d9-5145a898b73d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/62063914-1f6c-4e07-b0d9-5145a898b73d", "name": "62063914-1f6c-4e07-b0d9-5145a898b73d", "status": "Failed", "startTime": "2026-08-25T16:23:49.237499205Z", "endTime": "2026-08-25T16:42:58.795937162Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-xhfxlf4rcz52/cluster-candidate-4-21-rvhxrv should provision Unexpected error: <fmt.wrapError \| 0xc000f54d40>: failed to create HCP cluster cluster-candidate-4-21-rvhxrv: failed waiting for cluster="cluster-candidate-4-21-rvhxrv" in resourcegroup="rg-candidate-4-21-xhfxlf4rcz52" to finish creating: GET https://rp.j0599680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/62063914-1f6c-4e07-b0d9-5145a898b73d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/62063914-1f6c-4e07-b0d9-5145a898b73d", "name": "62063914-1f6c-4e07-b0d9-5145a898b73d", "status": "Failed", "startTime": "2026-08-25T16:23:49.237499205Z", "endTime": "2026-08-25T16:42:58.795937162Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator]; provider Microsoft.RedHatOpenShift 15:16:27 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-c4b48j" with version 4.21.29 on candidate channel Unexpected error: <fmt.wrapError \| 0xc001892580>: failed to create HCP cluster cluster-zstream-4-21-c4b48j: failed waiting for cluster="cluster-zstream-4-21-c4b48j" in resourcegroup="rg-zstream-upgrade-4-21-tjwbnczmk66w" to finish creating: GET https://rp.j0599680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5dd2277c-c818-44a4-91cb-db69efb21df4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5dd2277c-c818-44a4-91cb-db69efb21df4", "name": "5dd2277c-c818-44a4-91cb-db69efb21df4", "status": "Failed", "startTime": "2026-08-25T16:24:22.258032156Z", "endTime": "2026-08-25T16:43:28.791772827Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-c4b48j" with version 4.21.29 on candidate channel Unexpected error: <fmt.wrapError \| 0xc001892580>: failed to create HCP cluster cluster-zstream-4-21-c4b48j: failed waiting for cluster="cluster-zstream-4-21-c4b48j" in resourcegroup="rg-zstream-upgrade-4-21-tjwbnczmk66w" to finish creating: GET https://rp.j0599680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5dd2277c-c818-44a4-91cb-db69efb21df4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5dd2277c-c818-44a4-91cb-db69efb21df4", "name": "5dd2277c-c818-44a4-91cb-db69efb21df4", "status": "Failed", "startTime": "2026-08-25T16:24:22.258032156Z", "endTime": "2026-08-25T16:43:28.791772827Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift 15:16:27 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc00138c280>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-s8fx96sjztzk" to finish creating: GET https://rp.j0599680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b66bae1c-c22e-47b7-adb2-c5dc89eaa631 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b66bae1c-c22e-47b7-adb2-c5dc89eaa631", "name": "b66bae1c-c22e-47b7-adb2-c5dc89eaa631", "status": "Failed", "startTime": "2026-08-25T16:23:49.655363772Z", "endTime": "2026-08-25T16:42:58.742685687Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc00138c280>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-s8fx96sjztzk" to finish creating: GET https://rp.j0599680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b66bae1c-c22e-47b7-adb2-c5dc89eaa631 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b66bae1c-c22e-47b7-adb2-c5dc89eaa631", "name": "b66bae1c-c22e-47b7-adb2-c5dc89eaa631", "status": "Failed", "startTime": "2026-08-25T16:23:49.655363772Z", "endTime": "2026-08-25T16:42:58.742685687Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift 15:16:27 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--sgwjdjgmhg59/cluster-hs-pre-28tpcf should provision Unexpected error: <fmt.wrapError \| 0xc0010fc840>: failed to create HCP cluster cluster-hs-pre-28tpcf: failed waiting for cluster="cluster-hs-pre-28tpcf" in resourcegroup="rg-hs-pre--sgwjdjgmhg59" to finish creating: GET https://rp.j0599680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd3092ff-ff71-4241-9dfd-bed54a05f55a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd3092ff-ff71-4241-9dfd-bed54a05f55a", "name": "bd3092ff-ff71-4241-9dfd-bed54a05f55a", "status": "Failed", "startTime": "2026-08-25T16:23:47.826894918Z", "endTime": "2026-08-25T16:42:48.770504975Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--sgwjdjgmhg59/cluster-hs-pre-28tpcf should provision Unexpected error: <fmt.wrapError \| 0xc0010fc840>: failed to create HCP cluster cluster-hs-pre-28tpcf: failed waiting for cluster="cluster-hs-pre-28tpcf" in resourcegroup="rg-hs-pre--sgwjdjgmhg59" to finish creating: GET https://rp.j0599680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd3092ff-ff71-4241-9dfd-bed54a05f55a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd3092ff-ff71-4241-9dfd-bed54a05f55a", "name": "bd3092ff-ff71-4241-9dfd-bed54a05f55a", "status": "Failed", "startTime": "2026-08-25T16:23:47.826894918Z", "endTime": "2026-08-25T16:42:48.770504975Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred
15:16:24	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 3c81c74	alert, e2e	failed	⚠#6645 (open)	21	e2e (19) Cluster provisioning failed 15:16:24 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 15:16:24 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc000f71140>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-7fhrl8nsqhmm" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9ec8537f-8ee3-4b4f-990c-d50e9ecb7497 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9ec8537f-8ee3-4b4f-990c-d50e9ecb7497", "name": "9ec8537f-8ee3-4b4f-990c-d50e9ecb7497", "status": "Failed", "startTime": "2026-08-25T16:21:28.727590226Z", "endTime": "2026-08-25T16:40:28.622162047Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc000f71140>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-7fhrl8nsqhmm" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9ec8537f-8ee3-4b4f-990c-d50e9ecb7497 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9ec8537f-8ee3-4b4f-990c-d50e9ecb7497", "name": "9ec8537f-8ee3-4b4f-990c-d50e9ecb7497", "status": "Failed", "startTime": "2026-08-25T16:21:28.727590226Z", "endTime": "2026-08-25T16:40:28.622162047Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift 15:16:24 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:217]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0xc000e62940>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-sd6rcg2rjkrp" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ac0fd996-f807-4332-8f23-716715bbe1f9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ac0fd996-f807-4332-8f23-716715bbe1f9", "name": "ac0fd996-f807-4332-8f23-716715bbe1f9", "status": "Failed", "startTime": "2026-08-25T16:21:28.235084078Z", "endTime": "2026-08-25T16:40:28.60703914Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:217]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0xc000e62940>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-sd6rcg2rjkrp" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ac0fd996-f807-4332-8f23-716715bbe1f9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ac0fd996-f807-4332-8f23-716715bbe1f9", "name": "ac0fd996-f807-4332-8f23-716715bbe1f9", "status": "Failed", "startTime": "2026-08-25T16:21:28.235084078Z", "endTime": "2026-08-25T16:40:28.60703914Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift 15:16:24 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-zgjbj5 with version 4.20.35 Unexpected error: <fmt.wrapError \| 0xc001b1e160>: failed to create HCP cluster np-version-upgrade-cluster-zgjbj5: failed waiting for cluster="np-version-upgrade-cluster-zgjbj5" in resourcegroup="rg-np-version-upgrade-zgjbj5-5lcrgtsl9x9p" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ccd91084-1a3a-4d2a-a6a7-9f3e0b14e716 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ccd91084-1a3a-4d2a-a6a7-9f3e0b14e716", "name": "ccd91084-1a3a-4d2a-a6a7-9f3e0b14e716", "status": "Failed", "startTime": "2026-08-25T16:20:59.293835394Z", "endTime": "2026-08-25T16:40:08.618237862Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-zgjbj5 with version 4.20.35 Unexpected error: <fmt.wrapError \| 0xc001b1e160>: failed to create HCP cluster np-version-upgrade-cluster-zgjbj5: failed waiting for cluster="np-version-upgrade-cluster-zgjbj5" in resourcegroup="rg-np-version-upgrade-zgjbj5-5lcrgtsl9x9p" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ccd91084-1a3a-4d2a-a6a7-9f3e0b14e716 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ccd91084-1a3a-4d2a-a6a7-9f3e0b14e716", "name": "ccd91084-1a3a-4d2a-a6a7-9f3e0b14e716", "status": "Failed", "startTime": "2026-08-25T16:20:59.293835394Z", "endTime": "2026-08-25T16:40:08.618237862Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 15:16:24 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000f54ca0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-jm7j95xkmcf2" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1fd76614-fbf5-4f0a-8889-fc415cb6c440 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1fd76614-fbf5-4f0a-8889-fc415cb6c440", "name": "1fd76614-fbf5-4f0a-8889-fc415cb6c440", "status": "Failed", "startTime": "2026-08-25T16:22:37.538637853Z", "endTime": "2026-08-25T16:41:38.64255925Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000f54ca0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-jm7j95xkmcf2" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1fd76614-fbf5-4f0a-8889-fc415cb6c440 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1fd76614-fbf5-4f0a-8889-fc415cb6c440", "name": "1fd76614-fbf5-4f0a-8889-fc415cb6c440", "status": "Failed", "startTime": "2026-08-25T16:22:37.538637853Z", "endTime": "2026-08-25T16:41:38.64255925Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-controller-manager]; provider Microsoft.RedHatOpenShift 15:16:24 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-fnczl99xh8kl/cluster-candidate-4-21-jscsmk should provision Unexpected error: <fmt.wrapError \| 0xc00097d140>: failed to create HCP cluster cluster-candidate-4-21-jscsmk: failed waiting for cluster="cluster-candidate-4-21-jscsmk" in resourcegroup="rg-candidate-4-21-fnczl99xh8kl" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c896c3fa-e9be-48ff-abf7-d53fe9674288 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c896c3fa-e9be-48ff-abf7-d53fe9674288", "name": "c896c3fa-e9be-48ff-abf7-d53fe9674288", "status": "Failed", "startTime": "2026-08-25T16:21:27.440763508Z", "endTime": "2026-08-25T16:40:28.603771658Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-fnczl99xh8kl/cluster-candidate-4-21-jscsmk should provision Unexpected error: <fmt.wrapError \| 0xc00097d140>: failed to create HCP cluster cluster-candidate-4-21-jscsmk: failed waiting for cluster="cluster-candidate-4-21-jscsmk" in resourcegroup="rg-candidate-4-21-fnczl99xh8kl" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c896c3fa-e9be-48ff-abf7-d53fe9674288 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c896c3fa-e9be-48ff-abf7-d53fe9674288", "name": "c896c3fa-e9be-48ff-abf7-d53fe9674288", "status": "Failed", "startTime": "2026-08-25T16:21:27.440763508Z", "endTime": "2026-08-25T16:40:28.603771658Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator, cluster-policy-controller, cluster-storage-operator, dns-operator, ingress-operator, redhat-marketplace-catalog, redhat-operators-catalog]; provider Microsoft.RedHatOpenShift 15:16:24 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc0018da440>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-qd5fftk5xqnx" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3c61f442-7e7a-4287-a096-d40849d2499b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3c61f442-7e7a-4287-a096-d40849d2499b", "name": "3c61f442-7e7a-4287-a096-d40849d2499b", "status": "Failed", "startTime": "2026-08-25T16:21:37.077702612Z", "endTime": "2026-08-25T16:40:38.597952289Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, ingress-operator, dns-operator, cluster-image-registry-operator, redhat-operators-catalog, redhat-marketplace-catalog, cluster-storage-operator; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 2 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc0018da440>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-qd5fftk5xqnx" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3c61f442-7e7a-4287-a096-d40849d2499b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3c61f442-7e7a-4287-a096-d40849d2499b", "name": "3c61f442-7e7a-4287-a096-d40849d2499b", "status": "Failed", "startTime": "2026-08-25T16:21:37.077702612Z", "endTime": "2026-08-25T16:40:38.597952289Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, ingress-operator, dns-operator, cluster-image-registry-operator, redhat-operators-catalog, redhat-marketplace-catalog, cluster-storage-operator; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 2 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 15:16:24 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc000ee2ca0>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-bb6s68gx7kh6" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9561d49c-5877-4d79-855f-4df08a7e3fae -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9561d49c-5877-4d79-855f-4df08a7e3fae", "name": "9561d49c-5877-4d79-855f-4df08a7e3fae", "status": "Failed", "startTime": "2026-08-25T16:21:40.989997281Z", "endTime": "2026-08-25T16:40:48.583774176Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, dns-operator, olm-operator, cluster-version-operator, hosted-cluster-config-operator, cluster-network-operator, ingress-operator, csi-snapshot-controller-operator, catalog-operator, cluster-policy-controller, packageserver, community-operators-catalog, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 2 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc000ee2ca0>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-bb6s68gx7kh6" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9561d49c-5877-4d79-855f-4df08a7e3fae -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9561d49c-5877-4d79-855f-4df08a7e3fae", "name": "9561d49c-5877-4d79-855f-4df08a7e3fae", "status": "Failed", "startTime": "2026-08-25T16:21:40.989997281Z", "endTime": "2026-08-25T16:40:48.583774176Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, dns-operator, olm-operator, cluster-version-operator, hosted-cluster-config-operator, cluster-network-operator, ingress-operator, csi-snapshot-controller-operator, catalog-operator, cluster-policy-controller, packageserver, community-operators-catalog, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 2 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 15:16:24 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-grhvwwgnmxfw/cluster-candidate-4-20-s6k958 should provision Unexpected error: <fmt.wrapError \| 0xc00111c140>: failed to create HCP cluster cluster-candidate-4-20-s6k958: failed waiting for cluster="cluster-candidate-4-20-s6k958" in resourcegroup="rg-candidate-4-20-grhvwwgnmxfw" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d2555a7c-e636-4588-b48a-b1225e2d8f06 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d2555a7c-e636-4588-b48a-b1225e2d8f06", "name": "d2555a7c-e636-4588-b48a-b1225e2d8f06", "status": "Failed", "startTime": "2026-08-25T16:21:38.527859766Z", "endTime": "2026-08-25T16:40:38.616570204Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ignition-server, ignition-server-proxy, certified-operators-catalog, catalog-operator, control-plane-pki-operator, kube-controller-manager, kube-scheduler, cluster-network-operator, cluster-node-tuning-operator, cluster-version-operator, dns-operator, cluster-image-registry-operator, konnectivity-agent, redhat-operators-catalog, machine-approver, openshift-apiserver, openshift-controller-manager, ingress-operator, community-operators-catalog, cluster-policy-controller, openshift-route-controller-manager, cluster-storage-operator, olm-operator, azure-cloud-controller-manager, packageserver, hosted-cluster-config-operator, csi-snapshot-controller-operator, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas, konnectivity-agent deployment has 2 unavailable replicas, kube-controller-manager deployment has 2 unavailable replicas, kube-scheduler deployment has 2 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 3 unavailable replicas, packageserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-grhvwwgnmxfw/cluster-candidate-4-20-s6k958 should provision Unexpected error: <fmt.wrapError \| 0xc00111c140>: failed to create HCP cluster cluster-candidate-4-20-s6k958: failed waiting for cluster="cluster-candidate-4-20-s6k958" in resourcegroup="rg-candidate-4-20-grhvwwgnmxfw" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d2555a7c-e636-4588-b48a-b1225e2d8f06 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d2555a7c-e636-4588-b48a-b1225e2d8f06", "name": "d2555a7c-e636-4588-b48a-b1225e2d8f06", "status": "Failed", "startTime": "2026-08-25T16:21:38.527859766Z", "endTime": "2026-08-25T16:40:38.616570204Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ignition-server, ignition-server-proxy, certified-operators-catalog, catalog-operator, control-plane-pki-operator, kube-controller-manager, kube-scheduler, cluster-network-operator, cluster-node-tuning-operator, cluster-version-operator, dns-operator, cluster-image-registry-operator, konnectivity-agent, redhat-operators-catalog, machine-approver, openshift-apiserver, openshift-controller-manager, ingress-operator, community-operators-catalog, cluster-policy-controller, openshift-route-controller-manager, cluster-storage-operator, olm-operator, azure-cloud-controller-manager, packageserver, hosted-cluster-config-operator, csi-snapshot-controller-operator, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas, konnectivity-agent deployment has 2 unavailable replicas, kube-controller-manager deployment has 2 unavailable replicas, kube-scheduler deployment has 2 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 3 unavailable replicas, packageserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift 15:16:24 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-prc4m6" with version 4.20.34 on candidate channel Unexpected error: <fmt.wrapError \| 0xc001c9c3c0>: failed to create HCP cluster cluster-zstream-4-20-prc4m6: failed waiting for cluster="cluster-zstream-4-20-prc4m6" in resourcegroup="rg-zstream-upgrade-4-20-74gcx95r2n24" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7a916a4f-de9b-4e7b-9d68-7d520bc7ba74 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7a916a4f-de9b-4e7b-9d68-7d520bc7ba74", "name": "7a916a4f-de9b-4e7b-9d68-7d520bc7ba74", "status": "Failed", "startTime": "2026-08-25T16:20:56.871797957Z", "endTime": "2026-08-25T16:39:58.557618251Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-prc4m6" with version 4.20.34 on candidate channel Unexpected error: <fmt.wrapError \| 0xc001c9c3c0>: failed to create HCP cluster cluster-zstream-4-20-prc4m6: failed waiting for cluster="cluster-zstream-4-20-prc4m6" in resourcegroup="rg-zstream-upgrade-4-20-74gcx95r2n24" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7a916a4f-de9b-4e7b-9d68-7d520bc7ba74 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7a916a4f-de9b-4e7b-9d68-7d520bc7ba74", "name": "7a916a4f-de9b-4e7b-9d68-7d520bc7ba74", "status": "Failed", "startTime": "2026-08-25T16:20:56.871797957Z", "endTime": "2026-08-25T16:39:58.557618251Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift 15:16:24 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-6nrgxcz8znbm/cluster-candidate-5-0-bz6rh7 should provision Unexpected error: <fmt.wrapError \| 0xc001044860>: failed to create HCP cluster cluster-candidate-5-0-bz6rh7: failed waiting for cluster="cluster-candidate-5-0-bz6rh7" in resourcegroup="rg-candidate-5-0-6nrgxcz8znbm" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c0e826bb-86ad-484d-940a-a69d7c037731 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c0e826bb-86ad-484d-940a-a69d7c037731", "name": "c0e826bb-86ad-484d-940a-a69d7c037731", "status": "Failed", "startTime": "2026-08-25T16:21:17.696794179Z", "endTime": "2026-08-25T16:40:18.610816463Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-6nrgxcz8znbm/cluster-candidate-5-0-bz6rh7 should provision Unexpected error: <fmt.wrapError \| 0xc001044860>: failed to create HCP cluster cluster-candidate-5-0-bz6rh7: failed waiting for cluster="cluster-candidate-5-0-bz6rh7" in resourcegroup="rg-candidate-5-0-6nrgxcz8znbm" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c0e826bb-86ad-484d-940a-a69d7c037731 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c0e826bb-86ad-484d-940a-a69d7c037731", "name": "c0e826bb-86ad-484d-940a-a69d7c037731", "status": "Failed", "startTime": "2026-08-25T16:21:17.696794179Z", "endTime": "2026-08-25T16:40:18.610816463Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 15:16:24 UTC · failed at e2e · test SRE can pause schedules to stop backup execution for an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000ff2780>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-6hgvwqwzvj5t" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2eba795a-0e4a-4269-8001-ab35fdb2081e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2eba795a-0e4a-4269-8001-ab35fdb2081e", "name": "2eba795a-0e4a-4269-8001-ab35fdb2081e", "status": "Failed", "startTime": "2026-08-25T16:21:39.227173208Z", "endTime": "2026-08-25T16:40:48.587074479Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000ff2780>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-6hgvwqwzvj5t" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2eba795a-0e4a-4269-8001-ab35fdb2081e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2eba795a-0e4a-4269-8001-ab35fdb2081e", "name": "2eba795a-0e4a-4269-8001-ab35fdb2081e", "status": "Failed", "startTime": "2026-08-25T16:21:39.227173208Z", "endTime": "2026-08-25T16:40:48.587074479Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 15:16:24 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc000def640>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-9f9z2nqvqrnx" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8aa85fcb-c56d-4b95-a957-349139c53706 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8aa85fcb-c56d-4b95-a957-349139c53706", "name": "8aa85fcb-c56d-4b95-a957-349139c53706", "status": "Failed", "startTime": "2026-08-25T16:23:31.739649334Z", "endTime": "2026-08-25T16:42:38.476116853Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-controller-manager, cluster-policy-controller, hosted-cluster-config-operator, cluster-node-tuning-operator, catalog-operator, redhat-operators-catalog, certified-operators-catalog, kube-scheduler, openshift-route-controller-manager, packageserver, openshift-apiserver, dns-operator, ingress-operator, cluster-version-operator, cluster-network-operator, olm-operator, redhat-marketplace-catalog, cluster-storage-operator, csi-snapshot-controller-operator, community-operators-catalog, openshift-controller-manager, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 2 unavailable replicas, packageserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc000def640>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-9f9z2nqvqrnx" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8aa85fcb-c56d-4b95-a957-349139c53706 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8aa85fcb-c56d-4b95-a957-349139c53706", "name": "8aa85fcb-c56d-4b95-a957-349139c53706", "status": "Failed", "startTime": "2026-08-25T16:23:31.739649334Z", "endTime": "2026-08-25T16:42:38.476116853Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-controller-manager, cluster-policy-controller, hosted-cluster-config-operator, cluster-node-tuning-operator, catalog-operator, redhat-operators-catalog, certified-operators-catalog, kube-scheduler, openshift-route-controller-manager, packageserver, openshift-apiserver, dns-operator, ingress-operator, cluster-version-operator, cluster-network-operator, olm-operator, redhat-marketplace-catalog, cluster-storage-operator, csi-snapshot-controller-operator, community-operators-catalog, openshift-controller-manager, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 2 unavailable replicas, packageserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 15:16:24 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260630preview and verify the ingress is internal · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc001002420>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-t2kb5kkgwfmt" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eef968d7-3821-4aeb-a3bc-7e141d9d4b7b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eef968d7-3821-4aeb-a3bc-7e141d9d4b7b", "name": "eef968d7-3821-4aeb-a3bc-7e141d9d4b7b", "status": "Failed", "startTime": "2026-08-25T16:21:26.515094904Z", "endTime": "2026-08-25T16:40:28.648310137Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc001002420>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-t2kb5kkgwfmt" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eef968d7-3821-4aeb-a3bc-7e141d9d4b7b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eef968d7-3821-4aeb-a3bc-7e141d9d4b7b", "name": "eef968d7-3821-4aeb-a3bc-7e141d9d4b7b", "status": "Failed", "startTime": "2026-08-25T16:21:26.515094904Z", "endTime": "2026-08-25T16:40:28.648310137Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift 15:16:24 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc000942ca0>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-kzttj9k8trpk" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/848dc16e-8d2f-492e-ad78-0e05ac46ed06 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/848dc16e-8d2f-492e-ad78-0e05ac46ed06", "name": "848dc16e-8d2f-492e-ad78-0e05ac46ed06", "status": "Failed", "startTime": "2026-08-25T16:22:26.59789954Z", "endTime": "2026-08-25T16:41:28.524110603Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, openshift-route-controller-manager, cluster-network-operator, ignition-server-proxy, cluster-policy-controller, hosted-cluster-config-operator, olm-collect-profiles, redhat-marketplace-catalog, cluster-autoscaler, control-plane-pki-operator, kube-controller-manager, cluster-node-tuning-operator, ingress-operator, packageserver, openshift-apiserver, cluster-image-registry-operator, olm-operator, openshift-controller-manager, catalog-operator, community-operators-catalog, ignition-server, redhat-operators-catalog, konnectivity-agent, kube-scheduler, azure-cloud-controller-manager, certified-operators-catalog, machine-approver, cluster-storage-operator, dns-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc000942ca0>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-kzttj9k8trpk" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/848dc16e-8d2f-492e-ad78-0e05ac46ed06 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/848dc16e-8d2f-492e-ad78-0e05ac46ed06", "name": "848dc16e-8d2f-492e-ad78-0e05ac46ed06", "status": "Failed", "startTime": "2026-08-25T16:22:26.59789954Z", "endTime": "2026-08-25T16:41:28.524110603Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, openshift-route-controller-manager, cluster-network-operator, ignition-server-proxy, cluster-policy-controller, hosted-cluster-config-operator, olm-collect-profiles, redhat-marketplace-catalog, cluster-autoscaler, control-plane-pki-operator, kube-controller-manager, cluster-node-tuning-operator, ingress-operator, packageserver, openshift-apiserver, cluster-image-registry-operator, olm-operator, openshift-controller-manager, catalog-operator, community-operators-catalog, ignition-server, redhat-operators-catalog, konnectivity-agent, kube-scheduler, azure-cloud-controller-manager, certified-operators-catalog, machine-approver, cluster-storage-operator, dns-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift 15:16:24 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--cgdbmlmgbcbn/cluster-hs-pre-5rbdg9 should provision Unexpected error: <fmt.wrapError \| 0xc000f0a340>: failed to create HCP cluster cluster-hs-pre-5rbdg9: failed waiting for cluster="cluster-hs-pre-5rbdg9" in resourcegroup="rg-hs-pre--cgdbmlmgbcbn" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7cb95384-7668-4364-9a3d-af25ebeda917 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7cb95384-7668-4364-9a3d-af25ebeda917", "name": "7cb95384-7668-4364-9a3d-af25ebeda917", "status": "Failed", "startTime": "2026-08-25T16:21:27.709484961Z", "endTime": "2026-08-25T16:40:28.634231916Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--cgdbmlmgbcbn/cluster-hs-pre-5rbdg9 should provision Unexpected error: <fmt.wrapError \| 0xc000f0a340>: failed to create HCP cluster cluster-hs-pre-5rbdg9: failed waiting for cluster="cluster-hs-pre-5rbdg9" in resourcegroup="rg-hs-pre--cgdbmlmgbcbn" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7cb95384-7668-4364-9a3d-af25ebeda917 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7cb95384-7668-4364-9a3d-af25ebeda917", "name": "7cb95384-7668-4364-9a3d-af25ebeda917", "status": "Failed", "startTime": "2026-08-25T16:21:27.709484961Z", "endTime": "2026-08-25T16:40:28.634231916Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 15:16:24 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-t55dzpdntqx7/cluster-candidate-4-22-4rzl9g should provision Unexpected error: <fmt.wrapError \| 0xc001738140>: failed to create HCP cluster cluster-candidate-4-22-4rzl9g: failed waiting for cluster="cluster-candidate-4-22-4rzl9g" in resourcegroup="rg-candidate-4-22-t55dzpdntqx7" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6edcffe4-fe02-4abf-a4aa-6cc45f718fbb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6edcffe4-fe02-4abf-a4aa-6cc45f718fbb", "name": "6edcffe4-fe02-4abf-a4aa-6cc45f718fbb", "status": "Failed", "startTime": "2026-08-25T16:21:37.209310875Z", "endTime": "2026-08-25T16:40:38.614625988Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-t55dzpdntqx7/cluster-candidate-4-22-4rzl9g should provision Unexpected error: <fmt.wrapError \| 0xc001738140>: failed to create HCP cluster cluster-candidate-4-22-4rzl9g: failed waiting for cluster="cluster-candidate-4-22-4rzl9g" in resourcegroup="rg-candidate-4-22-t55dzpdntqx7" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6edcffe4-fe02-4abf-a4aa-6cc45f718fbb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6edcffe4-fe02-4abf-a4aa-6cc45f718fbb", "name": "6edcffe4-fe02-4abf-a4aa-6cc45f718fbb", "status": "Failed", "startTime": "2026-08-25T16:21:37.209310875Z", "endTime": "2026-08-25T16:40:38.614625988Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 15:16:24 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc0010429c0>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-27m57z2w4jzb" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5176ecee-a4c1-4981-902c-1cba33665ee8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5176ecee-a4c1-4981-902c-1cba33665ee8", "name": "5176ecee-a4c1-4981-902c-1cba33665ee8", "status": "Failed", "startTime": "2026-08-25T16:21:00.853975604Z", "endTime": "2026-08-25T16:40:08.609691152Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc0010429c0>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-27m57z2w4jzb" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5176ecee-a4c1-4981-902c-1cba33665ee8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5176ecee-a4c1-4981-902c-1cba33665ee8", "name": "5176ecee-a4c1-4981-902c-1cba33665ee8", "status": "Failed", "startTime": "2026-08-25T16:21:00.853975604Z", "endTime": "2026-08-25T16:40:08.609691152Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator]; provider Microsoft.RedHatOpenShift 15:16:24 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-hrhcgp" with version 5.0.0-ec.5 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000dce540>: failed to create HCP cluster cluster-zstream-5-0-hrhcgp: failed waiting for cluster="cluster-zstream-5-0-hrhcgp" in resourcegroup="rg-zstream-upgrade-5-0-78mnn44hkdrc" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b95d4f1a-9b50-422c-9b91-4f716507dd2d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b95d4f1a-9b50-422c-9b91-4f716507dd2d", "name": "b95d4f1a-9b50-422c-9b91-4f716507dd2d", "status": "Failed", "startTime": "2026-08-25T16:21:27.785135797Z", "endTime": "2026-08-25T16:40:28.593240311Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-hrhcgp" with version 5.0.0-ec.5 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000dce540>: failed to create HCP cluster cluster-zstream-5-0-hrhcgp: failed waiting for cluster="cluster-zstream-5-0-hrhcgp" in resourcegroup="rg-zstream-upgrade-5-0-78mnn44hkdrc" to finish creating: GET https://rp.j0672512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b95d4f1a-9b50-422c-9b91-4f716507dd2d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b95d4f1a-9b50-422c-9b91-4f716507dd2d", "name": "b95d4f1a-9b50-422c-9b91-4f716507dd2d", "status": "Failed", "startTime": "2026-08-25T16:21:27.785135797Z", "endTime": "2026-08-25T16:40:28.593240311Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (2) alert [hcp] KubePodNotReady fired 15:16:24 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 24 time(s) (13 unknown, 11 known) Firing 1: State: Resolved Started: 2026-08-25T16:37:02Z Ended: 2026-08-25T16:48:01Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0672512-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdn656b4osgukbth9n3lum5gp6jaj2n-j8n9u2x5g0l2u1m", pod="router-5694df8766-77fjb", severity="warning" Description: Pod ocm-arohcpci01-2sdn656b4osgukbth9n3lum5gp6jaj2n-j8n9u2x5g0l2u1m/router-5694df8766-77fjb has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-08-25T16:37:02Z Ended: 2026-08-25T16:45:04Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0672512-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdn61l3u29crvag5ueaf2jktpso8sgs-private-ingress", pod="etcd-2", severity="warning" Description: Pod ocm-arohcpci01-2sdn61l3u29crvag5ueaf2jktpso8sgs-private-ingress/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-08-25T16:38:02Z Ended: 2026-08-25T16:45:04Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0672512-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdn6h6baceee3dn4jbde3hiniipum3n-cilium-cluster", pod="router-6b4c99498f-zsdlc", severity="warning" Description: Pod ocm-arohcpci01-2sdn6h6baceee3dn4jbde3hiniipum3n-cilium-cluster/router-6b4c99498f-zsdlc has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-08-25T16:38:02Z Ended: 2026-08-25T16:53:06Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0672512-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdn62e5gdi5s69o2fmo3td2ihh4399i-p8m2k0b4j9z1w9l", pod="router-9cd49fbd4-5z2wr", severity="warning" Description: Pod ocm-arohcpci01-2sdn62e5gdi5s69o2fmo3td2ihh4399i-p8m2k0b4j9z1w9l/router-9cd49fbd4-5z2wr has been in a non-ready state for longer than 5 minutes. Firing 5: State: Resolved Started: 2026-08-25T16:38:02Z Ended: 2026-08-25T16:45:04Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0672512-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdn6h6baceee3dn4jbde3hiniipum3n-cilium-cluster", pod="etcd-1", severity="warning" Description: Pod ocm-arohcpci01-2sdn6h6baceee3dn4jbde3hiniipum3n-cilium-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 6: State: Resolved Started: 2026-08-25T16:38:02Z Ended: 2026-08-25T16:45:04Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0672512-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdn656b4osgukbth9n3lum5gp6jaj2n-j8n9u2x5g0l2u1m", pod="kube-apiserver-7fbbcc94d-fq9p6", severity="warning" Description: Pod ocm-arohcpci01-2sdn656b4osgukbth9n3lum5gp6jaj2n-j8n9u2x5g0l2u1m/kube-apiserver-7fbbcc94d-fq9p6 has been in a non-ready state for longer than 5 minutes. Firing 7: State: Resolved Started: 2026-08-25T16:38:02Z Ended: 2026-08-25T16:45:04Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0672512-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdn652p93biglu2ij3krnvkelsegk6n-x1n2h7r9m3o4t4m", pod="etcd-2", severity="warning" Description: Pod ocm-arohcpci01-2sdn652p93biglu2ij3krnvkelsegk6n-x1n2h7r9m3o4t4m/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 8: State: Resolved Started: 2026-08-25T16:39:01Z Ended: 2026-08-25T16:45:04Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0672512-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdn7153g2v0k31dooooj8u7k8mltn1a-agg-cluster", pod="router-675c4686fd-2xgdc", severity="warning" Description: Pod ocm-arohcpci01-2sdn7153g2v0k31dooooj8u7k8mltn1a-agg-cluster/router-675c4686fd-2xgdc has been in a non-ready state for longer than 5 minutes. Firing 9: State: Resolved Started: 2026-08-25T16:39:01Z Ended: 2026-08-25T16:45:04Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0672512-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdn62899n04b8kggs43s83ca16p2umu-oidc-wi-cluster", pod="multus-admission-controller-856d98f646-pw2k6", severity="warning" Description: Pod ocm-arohcpci01-2sdn62899n04b8kggs43s83ca16p2umu-oidc-wi-cluster/multus-admission-controller-856d98f646-pw2k6 has been in a non-ready state for longer than 5 minutes. Firing 10: State: Resolved Started: 2026-08-25T16:39:01Z Ended: 2026-08-25T16:45:04Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0672512-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdn62899n04b8kggs43s83ca16p2umu-oidc-wi-cluster", pod="network-node-identity-5486b5445d-g7cxx", severity="warning" Description: Pod ocm-arohcpci01-2sdn62899n04b8kggs43s83ca16p2umu-oidc-wi-cluster/network-node-identity-5486b5445d-g7cxx has been in a non-ready state for longer than 5 minutes. Firing 11: State: Resolved Started: 2026-08-25T16:39:01Z Ended: 2026-08-25T16:45:04Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0672512-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdn7153g2v0k31dooooj8u7k8mltn1a-agg-cluster", pod="etcd-2", severity="warning" Description: Pod ocm-arohcpci01-2sdn7153g2v0k31dooooj8u7k8mltn1a-agg-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 12: State: Resolved Started: 2026-08-25T16:39:01Z Ended: 2026-08-25T16:45:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0672512-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdn62899n04b8kggs43s83ca16p2umu-oidc-wi-cluster", pod="ovnkube-control-plane-844f7dbfcd-25zdb", severity="warning" Description: Pod ocm-arohcpci01-2sdn62899n04b8kggs43s83ca16p2umu-oidc-wi-cluster/ovnkube-control-plane-844f7dbfcd-25zdb has been in a non-ready state for longer than 5 minutes. Firing 13: State: Resolved Started: 2026-08-25T16:40:01Z Ended: 2026-08-25T16:51:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0672512-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdn5rat4libo1imashjsaa4uf5i00ah-i7z1w6u0o7q4z8r", pod="router-7c687f7dd9-wlx8z", severity="warning" Description: Pod ocm-arohcpci01-2sdn5rat4libo1imashjsaa4uf5i00ah-i7z1w6u0o7q4z8r/router-7c687f7dd9-wlx8z has been in a non-ready state for longer than 5 minutes. alert [svc] UJClusterProvisionErrors6h30m fired 15:16:24 UTC · failed at alert · test [aro-hcp-observability] [svc] alert UJClusterProvisionErrors6h30m does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-08-25T17:11:35Z Severity: Sev3 Labels: alertname="UJClusterProvisionErrors6h30m", cluster="ci01-j0672512-svc", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0672512/providers/microsoft.monitor/accounts/services-j0672512", severity="3", short_window="30m", slo="cluster-provision-errors" Description: More than 30% of cluster create (install) operations are in failed state sustained over 30 minutes, indicating a medium error budget burn (6x) that would exhaust the 95% SLO budget in ~28 hours.
15:16:22	pull-ci-Azure-ARO-HCP-main-e2e-parallel head b89541a	alert, e2e	failed	★#6670 (open)	6	e2e (5) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 15:16:22 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc001418640>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-2x96b4gl9jc2" to finish creating: GET https://rp.j3036032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5e909f63-9fd3-40b3-89a2-647eb7c84e5f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5e909f63-9fd3-40b3-89a2-647eb7c84e5f", "name": "5e909f63-9fd3-40b3-89a2-647eb7c84e5f", "status": "Failed", "startTime": "2026-08-25T16:14:50.211044644Z", "endTime": "2026-08-25T16:33:59.876273825Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc001418640>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-2x96b4gl9jc2" to finish creating: GET https://rp.j3036032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5e909f63-9fd3-40b3-89a2-647eb7c84e5f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5e909f63-9fd3-40b3-89a2-647eb7c84e5f", "name": "5e909f63-9fd3-40b3-89a2-647eb7c84e5f", "status": "Failed", "startTime": "2026-08-25T16:14:50.211044644Z", "endTime": "2026-08-25T16:33:59.876273825Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 15:16:22 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc000dd01e0>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-kw5c7cgnwx6r" to finish creating: GET https://rp.j3036032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8bc94cbd-7966-42ff-b2f3-8c3960b901a0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8bc94cbd-7966-42ff-b2f3-8c3960b901a0", "name": "8bc94cbd-7966-42ff-b2f3-8c3960b901a0", "status": "Failed", "startTime": "2026-08-25T16:14:34.084716184Z", "endTime": "2026-08-25T16:33:39.83687202Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc000dd01e0>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-kw5c7cgnwx6r" to finish creating: GET https://rp.j3036032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8bc94cbd-7966-42ff-b2f3-8c3960b901a0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8bc94cbd-7966-42ff-b2f3-8c3960b901a0", "name": "8bc94cbd-7966-42ff-b2f3-8c3960b901a0", "status": "Failed", "startTime": "2026-08-25T16:14:34.084716184Z", "endTime": "2026-08-25T16:33:39.83687202Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 15:16:22 UTC · failed at e2e · test FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc001200520>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-dzwjlx666575" to finish creating: GET https://rp.j3036032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/15fc6a54-c3a0-4f54-b9d1-9586d79644e9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/15fc6a54-c3a0-4f54-b9d1-9586d79644e9", "name": "15fc6a54-c3a0-4f54-b9d1-9586d79644e9", "status": "Failed", "startTime": "2026-08-25T16:13:38.854250497Z", "endTime": "2026-08-25T16:32:39.953721503Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc001200520>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-dzwjlx666575" to finish creating: GET https://rp.j3036032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/15fc6a54-c3a0-4f54-b9d1-9586d79644e9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/15fc6a54-c3a0-4f54-b9d1-9586d79644e9", "name": "15fc6a54-c3a0-4f54-b9d1-9586d79644e9", "status": "Failed", "startTime": "2026-08-25T16:13:38.854250497Z", "endTime": "2026-08-25T16:32:39.953721503Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 15:16:22 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc000f86580>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-qvdpwsdrhf5d" to finish creating: GET https://rp.j3036032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/11ae9dbb-7a03-4823-a826-d4717833e908 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/11ae9dbb-7a03-4823-a826-d4717833e908", "name": "11ae9dbb-7a03-4823-a826-d4717833e908", "status": "Failed", "startTime": "2026-08-25T16:13:50.746895077Z", "endTime": "2026-08-25T16:32:59.846114727Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc000f86580>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-qvdpwsdrhf5d" to finish creating: GET https://rp.j3036032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/11ae9dbb-7a03-4823-a826-d4717833e908 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/11ae9dbb-7a03-4823-a826-d4717833e908", "name": "11ae9dbb-7a03-4823-a826-d4717833e908", "status": "Failed", "startTime": "2026-08-25T16:13:50.746895077Z", "endTime": "2026-08-25T16:32:59.846114727Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 15:16:22 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260630preview and verify the ingress is internal · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc0013f6760>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-8qxvb7djrfqs" to finish creating: GET https://rp.j3036032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0ef3c075-0505-498f-acb7-bec851217620 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0ef3c075-0505-498f-acb7-bec851217620", "name": "0ef3c075-0505-498f-acb7-bec851217620", "status": "Failed", "startTime": "2026-08-25T16:13:28.807201061Z", "endTime": "2026-08-25T16:32:29.809496877Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc0013f6760>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-8qxvb7djrfqs" to finish creating: GET https://rp.j3036032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0ef3c075-0505-498f-acb7-bec851217620 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0ef3c075-0505-498f-acb7-bec851217620", "name": "0ef3c075-0505-498f-acb7-bec851217620", "status": "Failed", "startTime": "2026-08-25T16:13:28.807201061Z", "endTime": "2026-08-25T16:32:29.809496877Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 15:16:22 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 3 time(s) Firing 1: State: Resolved Started: 2026-08-25T16:25:19Z Ended: 2026-08-25T16:33:22Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3036032-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdn2a5elejp0cfku18d6rb3bv447o3c-private-ingress", pod="router-5bcb798bf8-zh5bc", severity="warning" Description: Pod ocm-arohcpci01-2sdn2a5elejp0cfku18d6rb3bv447o3c-private-ingress/router-5bcb798bf8-zh5bc has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-08-25T16:25:19Z Ended: 2026-08-25T16:39:20Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3036032-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdn2fvs43afb9nrsit8jlo6ct8g322g-v20260901", pod="router-6f97cb49b9-8rhvp", severity="warning" Description: Pod ocm-arohcpci01-2sdn2fvs43afb9nrsit8jlo6ct8g322g-v20260901/router-6f97cb49b9-8rhvp has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-08-25T16:25:19Z Ended: 2026-08-25T16:33:22Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3036032-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdn2dd3dlt36sr0dfi0c6siqogbnb8u-c4l2n5t0w7y1b4v", pod="router-545548bbbf-bbdk6", severity="warning" Description: Pod ocm-arohcpci01-2sdn2dd3dlt36sr0dfi0c6siqogbnb8u-c4l2n5t0w7y1b4v/router-545548bbbf-bbdk6 has been in a non-ready state for longer than 5 minutes.
15:10:32	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 7a93dd5	n/a	passed	#6672 (open)	0	n/a
14:46:18	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 0c4df62	unknown	failed	#6269 (closed)	1	non-artifact-backed failure (no junit artifacts)
14:43:04	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 5dfc8bf	alert, e2e	failed	#6676 (open)	4	e2e (3) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 14:43:04 UTC · failed at e2e · test FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc001342120>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-wttjbjlspm75" to finish creating: GET https://rp.j3905408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34bc5592-1de9-4751-8680-ffd928f8babf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34bc5592-1de9-4751-8680-ffd928f8babf", "name": "34bc5592-1de9-4751-8680-ffd928f8babf", "status": "Failed", "startTime": "2026-08-25T15:50:06.83845625Z", "endTime": "2026-08-25T16:09:10.978763078Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc001342120>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-wttjbjlspm75" to finish creating: GET https://rp.j3905408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34bc5592-1de9-4751-8680-ffd928f8babf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34bc5592-1de9-4751-8680-ffd928f8babf", "name": "34bc5592-1de9-4751-8680-ffd928f8babf", "status": "Failed", "startTime": "2026-08-25T15:50:06.83845625Z", "endTime": "2026-08-25T16:09:10.978763078Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 14:43:04 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--f9fqqjtpqkj4/cluster-hs-pre-b68c79 should provision Unexpected error: <fmt.wrapError \| 0xc000f02140>: failed to create HCP cluster cluster-hs-pre-b68c79: failed waiting for cluster="cluster-hs-pre-b68c79" in resourcegroup="rg-hs-pre--f9fqqjtpqkj4" to finish creating: GET https://rp.j3905408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c87a3dc5-3324-4e8e-8ece-0c46e4236baf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c87a3dc5-3324-4e8e-8ece-0c46e4236baf", "name": "c87a3dc5-3324-4e8e-8ece-0c46e4236baf", "status": "Failed", "startTime": "2026-08-25T15:49:44.606597271Z", "endTime": "2026-08-25T16:08:50.991532908Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--f9fqqjtpqkj4/cluster-hs-pre-b68c79 should provision Unexpected error: <fmt.wrapError \| 0xc000f02140>: failed to create HCP cluster cluster-hs-pre-b68c79: failed waiting for cluster="cluster-hs-pre-b68c79" in resourcegroup="rg-hs-pre--f9fqqjtpqkj4" to finish creating: GET https://rp.j3905408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c87a3dc5-3324-4e8e-8ece-0c46e4236baf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c87a3dc5-3324-4e8e-8ece-0c46e4236baf", "name": "c87a3dc5-3324-4e8e-8ece-0c46e4236baf", "status": "Failed", "startTime": "2026-08-25T15:49:44.606597271Z", "endTime": "2026-08-25T16:08:50.991532908Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 14:43:04 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260630preview and verify the ingress is internal · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc000ffe300>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-v8qzqqn7phmz" to finish creating: GET https://rp.j3905408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/92b9a3b5-8aba-4a43-86bf-ab3ad8b4a9e9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/92b9a3b5-8aba-4a43-86bf-ab3ad8b4a9e9", "name": "92b9a3b5-8aba-4a43-86bf-ab3ad8b4a9e9", "status": "Failed", "startTime": "2026-08-25T15:49:54.637795969Z", "endTime": "2026-08-25T16:09:01.068450014Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc000ffe300>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-v8qzqqn7phmz" to finish creating: GET https://rp.j3905408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/92b9a3b5-8aba-4a43-86bf-ab3ad8b4a9e9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/92b9a3b5-8aba-4a43-86bf-ab3ad8b4a9e9", "name": "92b9a3b5-8aba-4a43-86bf-ab3ad8b4a9e9", "status": "Failed", "startTime": "2026-08-25T15:49:54.637795969Z", "endTime": "2026-08-25T16:09:01.068450014Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 14:43:04 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 4 time(s) (3 unknown, 1 known) Firing 1: State: Resolved Started: 2026-08-25T16:05:58Z Ended: 2026-08-25T16:14:57Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3905408-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdmn678cjt1bvv40dsep03g6dp7ja6g-k1r6c2q1w6q2r1l", pod="router-697c8c9f66-6qpxz", severity="warning" Description: Pod ocm-arohcpci01-2sdmn678cjt1bvv40dsep03g6dp7ja6g-k1r6c2q1w6q2r1l/router-697c8c9f66-6qpxz has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-08-25T16:06:55Z Ended: 2026-08-25T16:13:55Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3905408-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdmnbtoke5te2abh8bef6jgas0h9teb-b2f9y5p2w9v8y9l", pod="router-5489fff6b-q2fpx", severity="warning" Description: Pod ocm-arohcpci01-2sdmnbtoke5te2abh8bef6jgas0h9teb-b2f9y5p2w9v8y9l/router-5489fff6b-q2fpx has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-08-25T16:06:55Z Ended: 2026-08-25T16:13:55Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3905408-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdmn8ppcpk07jkgi2itsdeesfjgkaq1-private-ingress", pod="router-78b97948b9-g9t8j", severity="warning" Description: Pod ocm-arohcpci01-2sdmn8ppcpk07jkgi2itsdeesfjgkaq1-private-ingress/router-78b97948b9-g9t8j has been in a non-ready state for longer than 5 minutes.
14:42:10	pull-ci-Azure-ARO-HCP-main-e2e-parallel head e0ff3c4	alert	failed	⚠#6614 (open)	2	alert (2) alert [infra] Cosmos DB Throttled Requests (429) High - arohcpci01-rp-j7405952 fired 14:42:10 UTC · failed at alert · test [aro-hcp-observability] [infra] alert Cosmos DB Throttled Requests (429) High - arohcpci01-rp-j7405952 does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-25T15:54:11Z Ended: 2026-08-25T16:01:15Z Severity: Sev3 Description: Cosmos DB returned more than 100 throttled requests (HTTP 429 - request rate too large) summed over a 5-minute window, evaluated every minute. Sustained 429s mean the workload is exceeding provisioned throughput - investigate hot partitions or increase provisioned throughput (RU/s). A low, steady rate of 429s that client SDKs retry transparently can be normal. https://learn.microsoft.com/azure/cosmos-db/troubleshoot-request-rate-too-large alert [svc] FrontendPathMedianLatency fired 14:42:10 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-25T15:53:01Z Ended: 2026-08-25T16:26:02Z Severity: Sev3 Labels: alertname="FrontendPathMedianLatency", cluster="ci01-j7405952-svc", component="frontend", method="put", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning" Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j7405952-svc).
14:41:34	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 5fdef9d	unknown	failed	#6364 (open)	1	non-artifact-backed failure (no junit artifacts)
14:41:22	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c6d4678	alert, e2e	failed	⚠#6629 (open)	9	e2e (7) ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; provider Microsoft.RedHatOpenShift 14:41:22 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-pg2742k6drr6/cluster-candidate-4-20-s9697v should provision Unexpected error: <fmt.wrapError \| 0xc00121c780>: failed to create HCP cluster cluster-candidate-4-20-s9697v: failed waiting for cluster="cluster-candidate-4-20-s9697v" in resourcegroup="rg-candidate-4-20-pg2742k6drr6" to finish creating: GET https://rp.j8947456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/18bd3e8a-96dd-4c6e-9e86-32f76567eee8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/18bd3e8a-96dd-4c6e-9e86-32f76567eee8", "name": "18bd3e8a-96dd-4c6e-9e86-32f76567eee8", "status": "Failed", "startTime": "2026-08-25T15:26:38.293191515Z", "endTime": "2026-08-25T15:45:40.878934392Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-pg2742k6drr6/cluster-candidate-4-20-s9697v should provision Unexpected error: <fmt.wrapError \| 0xc00121c780>: failed to create HCP cluster cluster-candidate-4-20-s9697v: failed waiting for cluster="cluster-candidate-4-20-s9697v" in resourcegroup="rg-candidate-4-20-pg2742k6drr6" to finish creating: GET https://rp.j8947456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/18bd3e8a-96dd-4c6e-9e86-32f76567eee8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/18bd3e8a-96dd-4c6e-9e86-32f76567eee8", "name": "18bd3e8a-96dd-4c6e-9e86-32f76567eee8", "status": "Failed", "startTime": "2026-08-25T15:26:38.293191515Z", "endTime": "2026-08-25T15:45:40.878934392Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 14:41:22 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc000b87360>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-75tdl88p84mj" to finish creating: GET https://rp.j8947456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/70638949-645c-4e6e-baf8-3c094f5f0cd7 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/70638949-645c-4e6e-baf8-3c094f5f0cd7", "name": "70638949-645c-4e6e-baf8-3c094f5f0cd7", "status": "Failed", "startTime": "2026-08-25T15:29:01.756170917Z", "endTime": "2026-08-25T15:48:10.941060109Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc000b87360>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-75tdl88p84mj" to finish creating: GET https://rp.j8947456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/70638949-645c-4e6e-baf8-3c094f5f0cd7 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/70638949-645c-4e6e-baf8-3c094f5f0cd7", "name": "70638949-645c-4e6e-baf8-3c094f5f0cd7", "status": "Failed", "startTime": "2026-08-25T15:29:01.756170917Z", "endTime": "2026-08-25T15:48:10.941060109Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift 14:41:22 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--fgqd2jbq4t7r/cluster-hs-pre-5w9whg should provision Unexpected error: <fmt.wrapError \| 0xc00149e980>: failed to create HCP cluster cluster-hs-pre-5w9whg: failed waiting for cluster="cluster-hs-pre-5w9whg" in resourcegroup="rg-hs-pre--fgqd2jbq4t7r" to finish creating: GET https://rp.j8947456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1d4456e8-aa9d-49fc-839c-5bca4f61aad1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1d4456e8-aa9d-49fc-839c-5bca4f61aad1", "name": "1d4456e8-aa9d-49fc-839c-5bca4f61aad1", "status": "Failed", "startTime": "2026-08-25T15:26:36.931048124Z", "endTime": "2026-08-25T15:45:40.904564768Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--fgqd2jbq4t7r/cluster-hs-pre-5w9whg should provision Unexpected error: <fmt.wrapError \| 0xc00149e980>: failed to create HCP cluster cluster-hs-pre-5w9whg: failed waiting for cluster="cluster-hs-pre-5w9whg" in resourcegroup="rg-hs-pre--fgqd2jbq4t7r" to finish creating: GET https://rp.j8947456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1d4456e8-aa9d-49fc-839c-5bca4f61aad1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1d4456e8-aa9d-49fc-839c-5bca4f61aad1", "name": "1d4456e8-aa9d-49fc-839c-5bca4f61aad1", "status": "Failed", "startTime": "2026-08-25T15:26:36.931048124Z", "endTime": "2026-08-25T15:45:40.904564768Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 14:41:22 UTC · failed at e2e · test SRE can pause schedules to stop backup execution for an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc001088e20>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-4qx7mnmm2gkt" to finish creating: GET https://rp.j8947456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a3436a18-cf8e-49ed-b7cb-d8646cec6de4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a3436a18-cf8e-49ed-b7cb-d8646cec6de4", "name": "a3436a18-cf8e-49ed-b7cb-d8646cec6de4", "status": "Failed", "startTime": "2026-08-25T15:26:37.927050597Z", "endTime": "2026-08-25T15:45:40.902757825Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc001088e20>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-4qx7mnmm2gkt" to finish creating: GET https://rp.j8947456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a3436a18-cf8e-49ed-b7cb-d8646cec6de4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a3436a18-cf8e-49ed-b7cb-d8646cec6de4", "name": "a3436a18-cf8e-49ed-b7cb-d8646cec6de4", "status": "Failed", "startTime": "2026-08-25T15:26:37.927050597Z", "endTime": "2026-08-25T15:45:40.902757825Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 14:41:22 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc0010d8480>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-gxzkjn22dzjv" to finish creating: GET https://rp.j8947456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0b524ffb-dbcb-4815-9127-6b0aaa409edf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0b524ffb-dbcb-4815-9127-6b0aaa409edf", "name": "0b524ffb-dbcb-4815-9127-6b0aaa409edf", "status": "Failed", "startTime": "2026-08-25T15:27:57.646849157Z", "endTime": "2026-08-25T15:47:00.925521895Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc0010d8480>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-gxzkjn22dzjv" to finish creating: GET https://rp.j8947456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0b524ffb-dbcb-4815-9127-6b0aaa409edf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0b524ffb-dbcb-4815-9127-6b0aaa409edf", "name": "0b524ffb-dbcb-4815-9127-6b0aaa409edf", "status": "Failed", "startTime": "2026-08-25T15:27:57.646849157Z", "endTime": "2026-08-25T15:47:00.925521895Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 14:41:22 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc00070c080>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-6vr9n8kcsq5h" to finish creating: GET https://rp.j8947456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4bad3fdc-14be-4d4e-8b59-9f0e4013950a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4bad3fdc-14be-4d4e-8b59-9f0e4013950a", "name": "4bad3fdc-14be-4d4e-8b59-9f0e4013950a", "status": "Failed", "startTime": "2026-08-25T15:27:19.498154616Z", "endTime": "2026-08-25T15:46:20.843517231Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc00070c080>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-6vr9n8kcsq5h" to finish creating: GET https://rp.j8947456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4bad3fdc-14be-4d4e-8b59-9f0e4013950a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4bad3fdc-14be-4d4e-8b59-9f0e4013950a", "name": "4bad3fdc-14be-4d4e-8b59-9f0e4013950a", "status": "Failed", "startTime": "2026-08-25T15:27:19.498154616Z", "endTime": "2026-08-25T15:46:20.843517231Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 14:41:22 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc00111e6a0>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-kmrf4dpx7tg9" to finish creating: GET https://rp.j8947456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/56f96552-e780-4e7b-a16e-c37885f1c49d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/56f96552-e780-4e7b-a16e-c37885f1c49d", "name": "56f96552-e780-4e7b-a16e-c37885f1c49d", "status": "Failed", "startTime": "2026-08-25T15:27:29.975055079Z", "endTime": "2026-08-25T15:46:30.914374064Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc00111e6a0>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-kmrf4dpx7tg9" to finish creating: GET https://rp.j8947456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/56f96552-e780-4e7b-a16e-c37885f1c49d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/56f96552-e780-4e7b-a16e-c37885f1c49d", "name": "56f96552-e780-4e7b-a16e-c37885f1c49d", "status": "Failed", "startTime": "2026-08-25T15:27:29.975055079Z", "endTime": "2026-08-25T15:46:30.914374064Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (2) alert [hcp] KubePodNotReady fired 14:41:22 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) (1 unknown, 1 known) Firing 1: State: Resolved Started: 2026-08-25T15:55:27Z Ended: 2026-08-25T16:01:29Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8947456-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdmcm4cjq1180vpa50bh0k8c7f1ruqb-private-kas", pod="router-787b99fdd9-wvbbf", severity="warning" Description: Pod ocm-arohcpci01-2sdmcm4cjq1180vpa50bh0k8c7f1ruqb-private-kas/router-787b99fdd9-wvbbf has been in a non-ready state for longer than 5 minutes. alert [infra] Cosmos DB Normalized RU Consumption High - arohcpci01-rp-j8947456 fired 14:41:22 UTC · failed at alert · test [aro-hcp-observability] [infra] alert Cosmos DB Normalized RU Consumption High - arohcpci01-rp-j8947456 does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-25T15:54:58Z Ended: 2026-08-25T16:00:59Z Severity: Sev3 Description: Cosmos DB normalized RU consumption is above 70% averaged over a 10-minute window, evaluated every minute. Investigate workload patterns or increase provisioned throughput. https://learn.microsoft.com/azure/cosmos-db/monitor-normalized-request-units
14:41:05	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 34de8d3	unknown	failed	#6070 (open)	1	non-artifact-backed failure (no junit artifacts)
14:38:17	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 4277637	alert, e2e	failed	⚠#6648 (open)	5	e2e (4) cluster entered terminal Failed state after role assignment deployment 14:38:17 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.001s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.001s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 14:38:17 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-sr4hjt444ntc/cluster-candidate-4-20-dvjvdg should provision Unexpected error: <fmt.wrapError \| 0xc0018fc660>: failed to create HCP cluster cluster-candidate-4-20-dvjvdg: failed waiting for cluster="cluster-candidate-4-20-dvjvdg" in resourcegroup="rg-candidate-4-20-sr4hjt444ntc" to finish creating: GET https://rp.j9625216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/17e06728-cb97-46af-9b26-ca0125fb2f5d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/17e06728-cb97-46af-9b26-ca0125fb2f5d", "name": "17e06728-cb97-46af-9b26-ca0125fb2f5d", "status": "Failed", "startTime": "2026-08-25T15:40:51.829519403Z", "endTime": "2026-08-25T15:59:58.300021131Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-sr4hjt444ntc/cluster-candidate-4-20-dvjvdg should provision Unexpected error: <fmt.wrapError \| 0xc0018fc660>: failed to create HCP cluster cluster-candidate-4-20-dvjvdg: failed waiting for cluster="cluster-candidate-4-20-dvjvdg" in resourcegroup="rg-candidate-4-20-sr4hjt444ntc" to finish creating: GET https://rp.j9625216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/17e06728-cb97-46af-9b26-ca0125fb2f5d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/17e06728-cb97-46af-9b26-ca0125fb2f5d", "name": "17e06728-cb97-46af-9b26-ca0125fb2f5d", "status": "Failed", "startTime": "2026-08-25T15:40:51.829519403Z", "endTime": "2026-08-25T15:59:58.300021131Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred failed to get latest install version for candidate channel: failed getting controlPlaneVersion: Get "<url> 14:38:17 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:143]: failed to get latest install version for candidate channel: failed getting controlPlaneVersion: Get "https://api.openshift.com/api/upgrades_info/graph?arch=multi&channel=candidate-4.20": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.129.22.77:37397->172.30.0.10:53: i/o timeout fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:143]: failed to get latest install version for candidate channel: failed getting controlPlaneVersion: Get "https://api.openshift.com/api/upgrades_info/graph?arch=multi&channel=candidate-4.20": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.129.22.77:37397->172.30.0.10:53: i/o timeout ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [kube-controller-manager]; provider Microsoft.RedHatOpenShift 14:38:17 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: failed to create HCP cluster np-update-nodes-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000ff8340>: failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-qhbwgqnksltm" to finish creating: GET https://rp.j9625216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/549edf52-d404-4f5d-b33e-34f0219e1814 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/549edf52-d404-4f5d-b33e-34f0219e1814", "name": "549edf52-d404-4f5d-b33e-34f0219e1814", "status": "Failed", "startTime": "2026-08-25T15:41:53.637919079Z", "endTime": "2026-08-25T16:00:58.269163772Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: failed to create HCP cluster np-update-nodes-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000ff8340>: failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-qhbwgqnksltm" to finish creating: GET https://rp.j9625216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/549edf52-d404-4f5d-b33e-34f0219e1814 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/549edf52-d404-4f5d-b33e-34f0219e1814", "name": "549edf52-d404-4f5d-b33e-34f0219e1814", "status": "Failed", "startTime": "2026-08-25T15:41:53.637919079Z", "endTime": "2026-08-25T16:00:58.269163772Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 14:38:17 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 20 time(s) (1 unknown, 19 known) Firing 1: State: Resolved Started: 2026-08-25T15:55:35Z Ended: 2026-08-25T16:14:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9625216-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdmj1cccodu4opnletdsdsq44jkjroi-a5s1d1v8k8o6t3d", pod="router-57477b9b98-d4bgr", severity="warning" Description: Pod ocm-arohcpci01-2sdmj1cccodu4opnletdsdsq44jkjroi-a5s1d1v8k8o6t3d/router-57477b9b98-d4bgr has been in a non-ready state for longer than 5 minutes.
14:37:54	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c54c328	provision	failed	#6661 (open)	1	provision (1) ERROR CODE: DeploymentNotFound 14:37:54 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Fleet.Registration/management/output (stamp=1) · suite step graph Show raw failure time=2026-08-25T15:21:22.056Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=management step=output stamp=1 description="Step output\n Kind: ARM\n Template: ../../dev-infrastructure/templates/output-mgmt.bicep\n Parameters: ../../dev-infrastructure/configurations/output-mgmt.tmpl.bicepparam" time=2026-08-25T15:21:22.495Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=management step=output stamp=1 time=2026-08-25T15:21:22.967Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=management step=output stamp=1 deployment=5e98ce680849833c628986a2e68efe7c2667cb97ae72f93416c33a4c5e66f0ac portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j6676864-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F5e98ce680849833c628986a2e68efe7c2667cb97ae72f93416c33a4c5e66f0ac time=2026-08-25T15:21:23.080Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=management step=output stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6676864-mgmt-1/providers/Microsoft.Resources/deployments/5e98ce680849833c628986a2e68efe7c2667cb97ae72f93416c33a4c5e66f0ac/operationStatuses/08584139356027184199\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DeploymentNotFound\",\n \"message\": \"Deployment '5e98ce680849833c628986a2e68efe7c2667cb97ae72f93416c33a4c5e66f0ac' could not be found.\"\n }\n}\n--------------------------------------------------------------------------------\n"
14:29:37	pull-ci-Azure-ARO-HCP-main-e2e-parallel head ca40fcb	alert	failed	#6658 (open)	2	alert (2) alert [hcp] KubePodNotReady fired 14:29:37 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-25T15:54:41Z Ended: 2026-08-25T16:00:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0144640-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdmbg6k1nbhvearl4c6d0hnook59uvs-v20260901", pod="openshift-apiserver-79fd97c87b-6mplf", severity="warning" Description: Pod ocm-arohcpci01-2sdmbg6k1nbhvearl4c6d0hnook59uvs-v20260901/openshift-apiserver-79fd97c87b-6mplf has been in a non-ready state for longer than 5 minutes. alert [svc] FleetControllerRetryHotLoop fired 14:29:37 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FleetControllerRetryHotLoop does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-25T15:28:49Z Ended: 2026-08-25T15:35:49Z Severity: Sev3 Labels: alertname="FleetControllerRetryHotLoop", cluster="ci01-j0144640-svc", component="fleet", name="clustersserviceregistrationcontroller", severity="warning" Description: Fleet controller workqueue clustersserviceregistrationcontroller has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work.
14:27:19	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 515e34b	e2e	failed	#6583 (open)	1	e2e (1) ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 14:27:19 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc000cce280>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-rg4h7xfrjp8d" to finish creating: GET https://rp.j4940032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9e2c6a53-377f-4d0e-9bd7-168eb63a2802 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9e2c6a53-377f-4d0e-9bd7-168eb63a2802", "name": "9e2c6a53-377f-4d0e-9bd7-168eb63a2802", "status": "Failed", "startTime": "2026-08-25T15:24:41.72360651Z", "endTime": "2026-08-25T15:43:49.738296088Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, csi-snapshot-controller-operator, redhat-marketplace-catalog, openshift-controller-manager, openshift-route-controller-manager, cluster-version-operator, cluster-storage-operator, ingress-operator, hosted-cluster-config-operator, packageserver, cluster-node-tuning-operator, olm-operator, redhat-operators-catalog, community-operators-catalog, dns-operator, catalog-operator, certified-operators-catalog, cluster-policy-controller, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc000cce280>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-rg4h7xfrjp8d" to finish creating: GET https://rp.j4940032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9e2c6a53-377f-4d0e-9bd7-168eb63a2802 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9e2c6a53-377f-4d0e-9bd7-168eb63a2802", "name": "9e2c6a53-377f-4d0e-9bd7-168eb63a2802", "status": "Failed", "startTime": "2026-08-25T15:24:41.72360651Z", "endTime": "2026-08-25T15:43:49.738296088Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, csi-snapshot-controller-operator, redhat-marketplace-catalog, openshift-controller-manager, openshift-route-controller-manager, cluster-version-operator, cluster-storage-operator, ingress-operator, hosted-cluster-config-operator, packageserver, cluster-node-tuning-operator, olm-operator, redhat-operators-catalog, community-operators-catalog, dns-operator, catalog-operator, certified-operators-catalog, cluster-policy-controller, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred
14:26:35	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 5c0ce63	e2e	failed	#6669 (open)	1	e2e (1) ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift 14:26:35 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-vq9mz9ctfdtr/cluster-candidate-4-22-lq4rs9 should provision Unexpected error: <fmt.wrapError \| 0xc00183a120>: failed to create HCP cluster cluster-candidate-4-22-lq4rs9: failed waiting for cluster="cluster-candidate-4-22-lq4rs9" in resourcegroup="rg-candidate-4-22-vq9mz9ctfdtr" to finish creating: GET https://rp.j0800128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/476cd9df-c870-4d67-9b4d-2d1a52e94c91 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/476cd9df-c870-4d67-9b4d-2d1a52e94c91", "name": "476cd9df-c870-4d67-9b4d-2d1a52e94c91", "status": "Failed", "startTime": "2026-08-25T15:26:35.691904584Z", "endTime": "2026-08-25T15:45:42.445862232Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-vq9mz9ctfdtr/cluster-candidate-4-22-lq4rs9 should provision Unexpected error: <fmt.wrapError \| 0xc00183a120>: failed to create HCP cluster cluster-candidate-4-22-lq4rs9: failed waiting for cluster="cluster-candidate-4-22-lq4rs9" in resourcegroup="rg-candidate-4-22-vq9mz9ctfdtr" to finish creating: GET https://rp.j0800128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/476cd9df-c870-4d67-9b4d-2d1a52e94c91 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/476cd9df-c870-4d67-9b4d-2d1a52e94c91", "name": "476cd9df-c870-4d67-9b4d-2d1a52e94c91", "status": "Failed", "startTime": "2026-08-25T15:26:35.691904584Z", "endTime": "2026-08-25T15:45:42.445862232Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred
14:14:30	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 75a66f4	unknown	failed	#6661 (open)	1	non-artifact-backed failure (no junit artifacts)
14:13:40	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c403afe	alert	failed	#6638 (open)	1	alert (1) alert [hcp] KubePodNotReady fired 14:13:40 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 20 time(s) (1 unknown, 19 known) Firing 1: State: Resolved Started: 2026-08-25T15:12:14Z Ended: 2026-08-25T15:18:19Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9320448-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlunpahc0163917kblhfvqqfmg5rbf-u9s4n8y1l2f2f0w", pod="cluster-network-operator-64b9458898-mnlcb", severity="warning" Description: Pod ocm-arohcpci01-2sdlunpahc0163917kblhfvqqfmg5rbf-u9s4n8y1l2f2f0w/cluster-network-operator-64b9458898-mnlcb has been in a non-ready state for longer than 5 minutes.
14:11:22	pull-ci-Azure-ARO-HCP-main-e2e-parallel head f0ff1a7	alert, e2e	failed	#6587 (open)	2	e2e (1) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 14:11:22 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc000de6180>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-w7q75thqw7pj" to finish creating: GET https://rp.j9425920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9b0c7885-5533-4919-9374-a469f0b9ceb9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9b0c7885-5533-4919-9374-a469f0b9ceb9", "name": "9b0c7885-5533-4919-9374-a469f0b9ceb9", "status": "Failed", "startTime": "2026-08-25T15:00:00.132611502Z", "endTime": "2026-08-25T15:19:06.963030567Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc000de6180>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-w7q75thqw7pj" to finish creating: GET https://rp.j9425920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9b0c7885-5533-4919-9374-a469f0b9ceb9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9b0c7885-5533-4919-9374-a469f0b9ceb9", "name": "9b0c7885-5533-4919-9374-a469f0b9ceb9", "status": "Failed", "startTime": "2026-08-25T15:00:00.132611502Z", "endTime": "2026-08-25T15:19:06.963030567Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 14:11:22 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 21 time(s) (1 unknown, 20 known) Firing 1: State: Resolved Started: 2026-08-25T15:29:40Z Ended: 2026-08-25T15:35:38Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9425920-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdm06nri3tscpmbts3s7sj19ggcvld4-ea-listlkvss6", pod="etcd-0", severity="warning" Description: Pod ocm-arohcpci01-2sdm06nri3tscpmbts3s7sj19ggcvld4-ea-listlkvss6/etcd-0 has been in a non-ready state for longer than 5 minutes.
14:10:01	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 7eff23c	e2e	failed	★#6575 (open)	8	e2e (8) ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 14:10:01 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-7bbblv" with version 4.20.34 on candidate channel Unexpected error: <fmt.wrapError \| 0xc0011f0940>: failed to create HCP cluster cluster-zstream-4-20-7bbblv: failed waiting for cluster="cluster-zstream-4-20-7bbblv" in resourcegroup="rg-zstream-upgrade-4-20-t8z5vnqflfbk" to finish creating: GET https://rp.j4024704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f6f56725-5efb-4b8e-89e7-68c89ae9fd73 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f6f56725-5efb-4b8e-89e7-68c89ae9fd73", "name": "f6f56725-5efb-4b8e-89e7-68c89ae9fd73", "status": "Failed", "startTime": "2026-08-25T14:53:45.994759307Z", "endTime": "2026-08-25T15:12:49.360735646Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-7bbblv" with version 4.20.34 on candidate channel Unexpected error: <fmt.wrapError \| 0xc0011f0940>: failed to create HCP cluster cluster-zstream-4-20-7bbblv: failed waiting for cluster="cluster-zstream-4-20-7bbblv" in resourcegroup="rg-zstream-upgrade-4-20-t8z5vnqflfbk" to finish creating: GET https://rp.j4024704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f6f56725-5efb-4b8e-89e7-68c89ae9fd73 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f6f56725-5efb-4b8e-89e7-68c89ae9fd73", "name": "f6f56725-5efb-4b8e-89e7-68c89ae9fd73", "status": "Failed", "startTime": "2026-08-25T14:53:45.994759307Z", "endTime": "2026-08-25T15:12:49.360735646Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator]; provider Microsoft.RedHatOpenShift 14:10:01 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc000f101e0>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-ghf78tzvlbsq" to finish creating: GET https://rp.j4024704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/72f94cb8-df3d-4852-ac78-087d67b17c38 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/72f94cb8-df3d-4852-ac78-087d67b17c38", "name": "72f94cb8-df3d-4852-ac78-087d67b17c38", "status": "Failed", "startTime": "2026-08-25T14:53:56.398337094Z", "endTime": "2026-08-25T15:12:59.401611971Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc000f101e0>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-ghf78tzvlbsq" to finish creating: GET https://rp.j4024704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/72f94cb8-df3d-4852-ac78-087d67b17c38 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/72f94cb8-df3d-4852-ac78-087d67b17c38", "name": "72f94cb8-df3d-4852-ac78-087d67b17c38", "status": "Failed", "startTime": "2026-08-25T14:53:56.398337094Z", "endTime": "2026-08-25T15:12:59.401611971Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift 14:10:01 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc0013d6380>: failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-xwfdq94j4nwz" to finish creating: GET https://rp.j4024704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c786f507-3037-418d-9e46-eab4b7d0f760 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c786f507-3037-418d-9e46-eab4b7d0f760", "name": "c786f507-3037-418d-9e46-eab4b7d0f760", "status": "Failed", "startTime": "2026-08-25T14:53:56.198919711Z", "endTime": "2026-08-25T15:12:59.311902322Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc0013d6380>: failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-xwfdq94j4nwz" to finish creating: GET https://rp.j4024704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c786f507-3037-418d-9e46-eab4b7d0f760 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c786f507-3037-418d-9e46-eab4b7d0f760", "name": "c786f507-3037-418d-9e46-eab4b7d0f760", "status": "Failed", "startTime": "2026-08-25T14:53:56.198919711Z", "endTime": "2026-08-25T15:12:59.311902322Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator, ignition-server-proxy, kube-controller-manager]; provider Microsoft.RedHatOpenShift 14:10:01 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: failed to create HCP cluster np-update-nodes-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000b67620>: failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-wgbj289pcnlt" to finish creating: GET https://rp.j4024704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/37f858c7-a07c-43e4-a020-589372bc5756 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/37f858c7-a07c-43e4-a020-589372bc5756", "name": "37f858c7-a07c-43e4-a020-589372bc5756", "status": "Failed", "startTime": "2026-08-25T14:54:06.405705309Z", "endTime": "2026-08-25T15:13:09.352514019Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: failed to create HCP cluster np-update-nodes-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000b67620>: failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-wgbj289pcnlt" to finish creating: GET https://rp.j4024704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/37f858c7-a07c-43e4-a020-589372bc5756 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/37f858c7-a07c-43e4-a020-589372bc5756", "name": "37f858c7-a07c-43e4-a020-589372bc5756", "status": "Failed", "startTime": "2026-08-25T14:54:06.405705309Z", "endTime": "2026-08-25T15:13:09.352514019Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift 14:10:01 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-pk8mtfhvgljp/cluster-candidate-4-20-bxb275 should provision Unexpected error: <fmt.wrapError \| 0xc0010dc8c0>: failed to create HCP cluster cluster-candidate-4-20-bxb275: failed waiting for cluster="cluster-candidate-4-20-bxb275" in resourcegroup="rg-candidate-4-20-pk8mtfhvgljp" to finish creating: GET https://rp.j4024704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7178f3ab-0cd5-4d88-98c6-7dbda7360f2e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7178f3ab-0cd5-4d88-98c6-7dbda7360f2e", "name": "7178f3ab-0cd5-4d88-98c6-7dbda7360f2e", "status": "Failed", "startTime": "2026-08-25T14:53:55.591878738Z", "endTime": "2026-08-25T15:12:59.329954081Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-pk8mtfhvgljp/cluster-candidate-4-20-bxb275 should provision Unexpected error: <fmt.wrapError \| 0xc0010dc8c0>: failed to create HCP cluster cluster-candidate-4-20-bxb275: failed waiting for cluster="cluster-candidate-4-20-bxb275" in resourcegroup="rg-candidate-4-20-pk8mtfhvgljp" to finish creating: GET https://rp.j4024704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7178f3ab-0cd5-4d88-98c6-7dbda7360f2e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7178f3ab-0cd5-4d88-98c6-7dbda7360f2e", "name": "7178f3ab-0cd5-4d88-98c6-7dbda7360f2e", "status": "Failed", "startTime": "2026-08-25T14:53:55.591878738Z", "endTime": "2026-08-25T15:12:59.329954081Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator]; provider Microsoft.RedHatOpenShift 14:10:01 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:442]: failed to create HCP cluster np-skip-minor-ms5drw at version 4.22.11 Unexpected error: <fmt.wrapError \| 0xc001050240>: failed to create HCP cluster np-skip-minor-ms5drw: failed waiting for cluster="np-skip-minor-ms5drw" in resourcegroup="rg-np-skip-minor-ms5drw-v9s5v98ch8vm" to finish creating: GET https://rp.j4024704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/07347ef0-e748-4346-97d6-c0ec7315b19c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/07347ef0-e748-4346-97d6-c0ec7315b19c", "name": "07347ef0-e748-4346-97d6-c0ec7315b19c", "status": "Failed", "startTime": "2026-08-25T14:53:58.520345438Z", "endTime": "2026-08-25T15:12:59.410612687Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:442]: failed to create HCP cluster np-skip-minor-ms5drw at version 4.22.11 Unexpected error: <fmt.wrapError \| 0xc001050240>: failed to create HCP cluster np-skip-minor-ms5drw: failed waiting for cluster="np-skip-minor-ms5drw" in resourcegroup="rg-np-skip-minor-ms5drw-v9s5v98ch8vm" to finish creating: GET https://rp.j4024704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/07347ef0-e748-4346-97d6-c0ec7315b19c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/07347ef0-e748-4346-97d6-c0ec7315b19c", "name": "07347ef0-e748-4346-97d6-c0ec7315b19c", "status": "Failed", "startTime": "2026-08-25T14:53:58.520345438Z", "endTime": "2026-08-25T15:12:59.410612687Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift 14:10:01 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-8zv2whbhlk5z/cluster-candidate-5-0-lxbbwb should provision Unexpected error: <fmt.wrapError \| 0xc00114a5c0>: failed to create HCP cluster cluster-candidate-5-0-lxbbwb: failed waiting for cluster="cluster-candidate-5-0-lxbbwb" in resourcegroup="rg-candidate-5-0-8zv2whbhlk5z" to finish creating: GET https://rp.j4024704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ca4022d-9521-44e9-8378-8f7a085e07b0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ca4022d-9521-44e9-8378-8f7a085e07b0", "name": "5ca4022d-9521-44e9-8378-8f7a085e07b0", "status": "Failed", "startTime": "2026-08-25T14:53:56.011376402Z", "endTime": "2026-08-25T15:12:59.566489384Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-8zv2whbhlk5z/cluster-candidate-5-0-lxbbwb should provision Unexpected error: <fmt.wrapError \| 0xc00114a5c0>: failed to create HCP cluster cluster-candidate-5-0-lxbbwb: failed waiting for cluster="cluster-candidate-5-0-lxbbwb" in resourcegroup="rg-candidate-5-0-8zv2whbhlk5z" to finish creating: GET https://rp.j4024704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ca4022d-9521-44e9-8378-8f7a085e07b0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ca4022d-9521-44e9-8378-8f7a085e07b0", "name": "5ca4022d-9521-44e9-8378-8f7a085e07b0", "status": "Failed", "startTime": "2026-08-25T14:53:56.011376402Z", "endTime": "2026-08-25T15:12:59.566489384Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift 14:10:01 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-9v2fm7" with version 4.22.10 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000abc760>: failed to create HCP cluster cluster-zstream-4-22-9v2fm7: failed waiting for cluster="cluster-zstream-4-22-9v2fm7" in resourcegroup="rg-zstream-upgrade-4-22-qzxxgjsj26qb" to finish creating: GET https://rp.j4024704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/02d2964a-7a05-4345-980d-51e9bdf2ffaf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/02d2964a-7a05-4345-980d-51e9bdf2ffaf", "name": "02d2964a-7a05-4345-980d-51e9bdf2ffaf", "status": "Failed", "startTime": "2026-08-25T14:54:16.06655817Z", "endTime": "2026-08-25T15:13:19.307227089Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-9v2fm7" with version 4.22.10 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000abc760>: failed to create HCP cluster cluster-zstream-4-22-9v2fm7: failed waiting for cluster="cluster-zstream-4-22-9v2fm7" in resourcegroup="rg-zstream-upgrade-4-22-qzxxgjsj26qb" to finish creating: GET https://rp.j4024704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/02d2964a-7a05-4345-980d-51e9bdf2ffaf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/02d2964a-7a05-4345-980d-51e9bdf2ffaf", "name": "02d2964a-7a05-4345-980d-51e9bdf2ffaf", "status": "Failed", "startTime": "2026-08-25T14:54:16.06655817Z", "endTime": "2026-08-25T15:13:19.307227089Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred
14:09:13	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 3c81c74	provision	failed	#6645 (open)	1	provision (1) ERROR CODE: DeploymentNotFound 14:09:13 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Fleet.Registration/management/output (stamp=2) · suite step graph Show raw failure time=2026-08-25T14:41:26.287Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=management step=output stamp=2 description="Step output\n Kind: ARM\n Template: ../../dev-infrastructure/templates/output-mgmt.bicep\n Parameters: ../../dev-infrastructure/configurations/output-mgmt.tmpl.bicepparam" time=2026-08-25T14:41:26.667Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=management step=output stamp=2 time=2026-08-25T14:41:27.045Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=management step=output stamp=2 deployment=3b44e853e3fa2f5ba695fe1eeadf1d38b957019eefef20ca1a49aa6b0c7a6f2b portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j6436096-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F3b44e853e3fa2f5ba695fe1eeadf1d38b957019eefef20ca1a49aa6b0c7a6f2b time=2026-08-25T14:41:27.116Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=management step=output stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6436096-mgmt-2/providers/Microsoft.Resources/deployments/3b44e853e3fa2f5ba695fe1eeadf1d38b957019eefef20ca1a49aa6b0c7a6f2b/operationStatuses/08584139379985620464\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DeploymentNotFound\",\n \"message\": \"Deployment '3b44e853e3fa2f5ba695fe1eeadf1d38b957019eefef20ca1a49aa6b0c7a6f2b' could not be found.\"\n }\n}\n--------------------------------------------------------------------------------\n"
14:01:22	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 53966db	alert, e2e	failed	⚠#6630 (open)	7	e2e (6) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator, ignition-server-proxy]; provider Microsoft.RedHatOpenShift 14:01:22 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000ee3660>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-pfd8fc762xff" to finish creating: GET https://rp.j9836800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c9934e2d-ec51-45a3-8cbd-60f4853ece87 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c9934e2d-ec51-45a3-8cbd-60f4853ece87", "name": "c9934e2d-ec51-45a3-8cbd-60f4853ece87", "status": "Failed", "startTime": "2026-08-25T14:47:46.321950941Z", "endTime": "2026-08-25T15:06:53.075223567Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000ee3660>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-pfd8fc762xff" to finish creating: GET https://rp.j9836800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c9934e2d-ec51-45a3-8cbd-60f4853ece87 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c9934e2d-ec51-45a3-8cbd-60f4853ece87", "name": "c9934e2d-ec51-45a3-8cbd-60f4853ece87", "status": "Failed", "startTime": "2026-08-25T14:47:46.321950941Z", "endTime": "2026-08-25T15:06:53.075223567Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 14:01:22 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-gnzmjzqw2kss/cluster-candidate-5-0-bbkwfd should provision Unexpected error: <fmt.wrapError \| 0xc000e0e640>: failed to create HCP cluster cluster-candidate-5-0-bbkwfd: failed waiting for cluster="cluster-candidate-5-0-bbkwfd" in resourcegroup="rg-candidate-5-0-gnzmjzqw2kss" to finish creating: GET https://rp.j9836800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4d991771-9053-4970-8d4f-39b2d38bc505 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4d991771-9053-4970-8d4f-39b2d38bc505", "name": "4d991771-9053-4970-8d4f-39b2d38bc505", "status": "Failed", "startTime": "2026-08-25T14:46:45.526857015Z", "endTime": "2026-08-25T15:05:53.073478915Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-gnzmjzqw2kss/cluster-candidate-5-0-bbkwfd should provision Unexpected error: <fmt.wrapError \| 0xc000e0e640>: failed to create HCP cluster cluster-candidate-5-0-bbkwfd: failed waiting for cluster="cluster-candidate-5-0-bbkwfd" in resourcegroup="rg-candidate-5-0-gnzmjzqw2kss" to finish creating: GET https://rp.j9836800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4d991771-9053-4970-8d4f-39b2d38bc505 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4d991771-9053-4970-8d4f-39b2d38bc505", "name": "4d991771-9053-4970-8d4f-39b2d38bc505", "status": "Failed", "startTime": "2026-08-25T14:46:45.526857015Z", "endTime": "2026-08-25T15:05:53.073478915Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift 14:01:22 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc00109e740>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-225vnhm84lkc" to finish creating: GET https://rp.j9836800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c7eb9a5b-8f23-46ff-8dff-b70b00b25966 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c7eb9a5b-8f23-46ff-8dff-b70b00b25966", "name": "c7eb9a5b-8f23-46ff-8dff-b70b00b25966", "status": "Failed", "startTime": "2026-08-25T14:48:06.4361502Z", "endTime": "2026-08-25T15:07:13.076236238Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc00109e740>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-225vnhm84lkc" to finish creating: GET https://rp.j9836800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c7eb9a5b-8f23-46ff-8dff-b70b00b25966 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c7eb9a5b-8f23-46ff-8dff-b70b00b25966", "name": "c7eb9a5b-8f23-46ff-8dff-b70b00b25966", "status": "Failed", "startTime": "2026-08-25T14:48:06.4361502Z", "endTime": "2026-08-25T15:07:13.076236238Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred cluster entered terminal Failed state after role assignment deployment 14:01:22 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.002s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.002s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator, ignition-server-proxy, kube-controller-manager]; provider Microsoft.RedHatOpenShift 14:01:22 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-dlh9fkj7trzd/cluster-candidate-4-21-59tf2h should provision Unexpected error: <fmt.wrapError \| 0xc000cba160>: failed to create HCP cluster cluster-candidate-4-21-59tf2h: failed waiting for cluster="cluster-candidate-4-21-59tf2h" in resourcegroup="rg-candidate-4-21-dlh9fkj7trzd" to finish creating: GET https://rp.j9836800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e26c41da-0377-42e1-a30e-6696aceb0e40 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e26c41da-0377-42e1-a30e-6696aceb0e40", "name": "e26c41da-0377-42e1-a30e-6696aceb0e40", "status": "Failed", "startTime": "2026-08-25T14:46:46.759513984Z", "endTime": "2026-08-25T15:05:52.992371742Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-dlh9fkj7trzd/cluster-candidate-4-21-59tf2h should provision Unexpected error: <fmt.wrapError \| 0xc000cba160>: failed to create HCP cluster cluster-candidate-4-21-59tf2h: failed waiting for cluster="cluster-candidate-4-21-59tf2h" in resourcegroup="rg-candidate-4-21-dlh9fkj7trzd" to finish creating: GET https://rp.j9836800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e26c41da-0377-42e1-a30e-6696aceb0e40 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e26c41da-0377-42e1-a30e-6696aceb0e40", "name": "e26c41da-0377-42e1-a30e-6696aceb0e40", "status": "Failed", "startTime": "2026-08-25T14:46:46.759513984Z", "endTime": "2026-08-25T15:05:52.992371742Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 14:01:22 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-ldcljf2ctfmx/cluster-candidate-4-22-bzcn4d should provision Unexpected error: <fmt.wrapError \| 0xc000ec0120>: failed to create HCP cluster cluster-candidate-4-22-bzcn4d: failed waiting for cluster="cluster-candidate-4-22-bzcn4d" in resourcegroup="rg-candidate-4-22-ldcljf2ctfmx" to finish creating: GET https://rp.j9836800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/26994133-5a3a-4dc6-8825-a517ba8b6492 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/26994133-5a3a-4dc6-8825-a517ba8b6492", "name": "26994133-5a3a-4dc6-8825-a517ba8b6492", "status": "Failed", "startTime": "2026-08-25T14:46:46.263967961Z", "endTime": "2026-08-25T15:05:53.038982489Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-ldcljf2ctfmx/cluster-candidate-4-22-bzcn4d should provision Unexpected error: <fmt.wrapError \| 0xc000ec0120>: failed to create HCP cluster cluster-candidate-4-22-bzcn4d: failed waiting for cluster="cluster-candidate-4-22-bzcn4d" in resourcegroup="rg-candidate-4-22-ldcljf2ctfmx" to finish creating: GET https://rp.j9836800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/26994133-5a3a-4dc6-8825-a517ba8b6492 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/26994133-5a3a-4dc6-8825-a517ba8b6492", "name": "26994133-5a3a-4dc6-8825-a517ba8b6492", "status": "Failed", "startTime": "2026-08-25T14:46:46.263967961Z", "endTime": "2026-08-25T15:05:53.038982489Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 14:01:22 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 24 time(s) (1 unknown, 23 known) Firing 1: State: Resolved Started: 2026-08-25T15:00:54Z Ended: 2026-08-25T15:20:53Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9836800-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlpln0oubfmm7l5rkf2udfse2o25ud-o6g6g1g5g3v3m3u", pod="router-7c5b776b9c-4765d", severity="warning" Description: Pod ocm-arohcpci01-2sdlpln0oubfmm7l5rkf2udfse2o25ud-o6g6g1g5g3v3m3u/router-7c5b776b9c-4765d has been in a non-ready state for longer than 5 minutes.
13:55:13	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 87c9d3c	alert, e2e	failed	#6662 (open)	5	e2e (4) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 13:55:13 UTC · failed at e2e · test SRE can pause schedules to stop backup execution for an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000d6d3c0>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-5rgd5mxghdpb" to finish creating: GET https://rp.j3839232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cf3f32cf-b93f-4884-bc0a-e3893898bf0c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cf3f32cf-b93f-4884-bc0a-e3893898bf0c", "name": "cf3f32cf-b93f-4884-bc0a-e3893898bf0c", "status": "Failed", "startTime": "2026-08-25T14:40:21.418735516Z", "endTime": "2026-08-25T14:59:24.842684505Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000d6d3c0>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-5rgd5mxghdpb" to finish creating: GET https://rp.j3839232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cf3f32cf-b93f-4884-bc0a-e3893898bf0c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cf3f32cf-b93f-4884-bc0a-e3893898bf0c", "name": "cf3f32cf-b93f-4884-bc0a-e3893898bf0c", "status": "Failed", "startTime": "2026-08-25T14:40:21.418735516Z", "endTime": "2026-08-25T14:59:24.842684505Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 13:55:13 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc000ee4d80>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-h2mbhr4mc5lp" to finish creating: GET https://rp.j3839232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c31876ee-87ff-43fd-a5d5-bd1cf0cac5a9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c31876ee-87ff-43fd-a5d5-bd1cf0cac5a9", "name": "c31876ee-87ff-43fd-a5d5-bd1cf0cac5a9", "status": "Failed", "startTime": "2026-08-25T14:40:53.432001681Z", "endTime": "2026-08-25T14:59:54.838826092Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc000ee4d80>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-h2mbhr4mc5lp" to finish creating: GET https://rp.j3839232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c31876ee-87ff-43fd-a5d5-bd1cf0cac5a9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c31876ee-87ff-43fd-a5d5-bd1cf0cac5a9", "name": "c31876ee-87ff-43fd-a5d5-bd1cf0cac5a9", "status": "Failed", "startTime": "2026-08-25T14:40:53.432001681Z", "endTime": "2026-08-25T14:59:54.838826092Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 13:55:13 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc00094e440>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-cjmmvcxplkm2" to finish creating: GET https://rp.j3839232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/729aa40b-70d3-4080-bbfb-0d103a608dbd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/729aa40b-70d3-4080-bbfb-0d103a608dbd", "name": "729aa40b-70d3-4080-bbfb-0d103a608dbd", "status": "Failed", "startTime": "2026-08-25T14:40:31.102654223Z", "endTime": "2026-08-25T14:59:34.736723772Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc00094e440>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-cjmmvcxplkm2" to finish creating: GET https://rp.j3839232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/729aa40b-70d3-4080-bbfb-0d103a608dbd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/729aa40b-70d3-4080-bbfb-0d103a608dbd", "name": "729aa40b-70d3-4080-bbfb-0d103a608dbd", "status": "Failed", "startTime": "2026-08-25T14:40:31.102654223Z", "endTime": "2026-08-25T14:59:34.736723772Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 13:55:13 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-dnw5jvjfdwmc/cluster-candidate-5-0-272q64 should provision Unexpected error: <fmt.wrapError \| 0xc000f74280>: failed to create HCP cluster cluster-candidate-5-0-272q64: failed waiting for cluster="cluster-candidate-5-0-272q64" in resourcegroup="rg-candidate-5-0-dnw5jvjfdwmc" to finish creating: GET https://rp.j3839232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e65c60e0-1cb1-4ac7-85e2-b7ecad902d21 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e65c60e0-1cb1-4ac7-85e2-b7ecad902d21", "name": "e65c60e0-1cb1-4ac7-85e2-b7ecad902d21", "status": "Failed", "startTime": "2026-08-25T14:40:31.01500763Z", "endTime": "2026-08-25T14:59:34.82035333Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-dnw5jvjfdwmc/cluster-candidate-5-0-272q64 should provision Unexpected error: <fmt.wrapError \| 0xc000f74280>: failed to create HCP cluster cluster-candidate-5-0-272q64: failed waiting for cluster="cluster-candidate-5-0-272q64" in resourcegroup="rg-candidate-5-0-dnw5jvjfdwmc" to finish creating: GET https://rp.j3839232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e65c60e0-1cb1-4ac7-85e2-b7ecad902d21 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e65c60e0-1cb1-4ac7-85e2-b7ecad902d21", "name": "e65c60e0-1cb1-4ac7-85e2-b7ecad902d21", "status": "Failed", "startTime": "2026-08-25T14:40:31.01500763Z", "endTime": "2026-08-25T14:59:34.82035333Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 13:55:13 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 15 time(s) (4 unknown, 11 known) Firing 1: State: Resolved Started: 2026-08-25T14:52:03Z Ended: 2026-08-25T15:12:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3839232-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlmo7baqls1gop29vgsct7kabvoth8-g0w4c9d3q6r2r1m", pod="router-974f5ddb-2dgs2", severity="warning" Description: Pod ocm-arohcpci01-2sdlmo7baqls1gop29vgsct7kabvoth8-g0w4c9d3q6r2r1m/router-974f5ddb-2dgs2 has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-08-25T14:52:03Z Ended: 2026-08-25T15:13:02Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3839232-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlmtn9ovk9tpsm9j2ft6snivi04oqe-a2p1p1g5j3i3f3u", pod="router-556f6fb4ff-nfxjj", severity="warning" Description: Pod ocm-arohcpci01-2sdlmtn9ovk9tpsm9j2ft6snivi04oqe-a2p1p1g5j3i3f3u/router-556f6fb4ff-nfxjj has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-08-25T14:53:02Z Ended: 2026-08-25T15:12:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3839232-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlmlash2aq785fmi7mq3rc7dfln3rc-k6a8t8m0v1f0j5x", pod="router-6bd4d54bc8-vsx2s", severity="warning" Description: Pod ocm-arohcpci01-2sdlmlash2aq785fmi7mq3rc7dfln3rc-k6a8t8m0v1f0j5x/router-6bd4d54bc8-vsx2s has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-08-25T14:55:02Z Ended: 2026-08-25T15:12:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3839232-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlmo1em3u37eqkj6qat0evo6kvf8qt-v20260901", pod="router-695b86fdc7-zr4fd", severity="warning" Description: Pod ocm-arohcpci01-2sdlmo1em3u37eqkj6qat0evo6kvf8qt-v20260901/router-695b86fdc7-zr4fd has been in a non-ready state for longer than 5 minutes.
13:52:37	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 0d881db	alert, e2e	failed	★#6677 (open)	5	e2e (4) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 13:52:37 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc000beb2a0>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-sn5zc7drh7f4" to finish creating: GET https://rp.j8970496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34aa8a23-7309-49a6-83bf-51c69ef9dc4e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34aa8a23-7309-49a6-83bf-51c69ef9dc4e", "name": "34aa8a23-7309-49a6-83bf-51c69ef9dc4e", "status": "Failed", "startTime": "2026-08-25T14:40:37.489505349Z", "endTime": "2026-08-25T14:59:39.545936275Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc000beb2a0>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-sn5zc7drh7f4" to finish creating: GET https://rp.j8970496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34aa8a23-7309-49a6-83bf-51c69ef9dc4e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34aa8a23-7309-49a6-83bf-51c69ef9dc4e", "name": "34aa8a23-7309-49a6-83bf-51c69ef9dc4e", "status": "Failed", "startTime": "2026-08-25T14:40:37.489505349Z", "endTime": "2026-08-25T14:59:39.545936275Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred cluster entered terminal Failed state after role assignment deployment 13:52:37 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.001s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.001s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 13:52:37 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-x6bg4m6clbjq/cluster-candidate-4-22-v8dh2g should provision Unexpected error: <fmt.wrapError \| 0xc001724ba0>: failed to create HCP cluster cluster-candidate-4-22-v8dh2g: failed waiting for cluster="cluster-candidate-4-22-v8dh2g" in resourcegroup="rg-candidate-4-22-x6bg4m6clbjq" to finish creating: GET https://rp.j8970496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8b6be2ed-087d-4652-a133-bf375421a904 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8b6be2ed-087d-4652-a133-bf375421a904", "name": "8b6be2ed-087d-4652-a133-bf375421a904", "status": "Failed", "startTime": "2026-08-25T14:40:17.286340435Z", "endTime": "2026-08-25T14:59:19.220463305Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-x6bg4m6clbjq/cluster-candidate-4-22-v8dh2g should provision Unexpected error: <fmt.wrapError \| 0xc001724ba0>: failed to create HCP cluster cluster-candidate-4-22-v8dh2g: failed waiting for cluster="cluster-candidate-4-22-v8dh2g" in resourcegroup="rg-candidate-4-22-x6bg4m6clbjq" to finish creating: GET https://rp.j8970496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8b6be2ed-087d-4652-a133-bf375421a904 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8b6be2ed-087d-4652-a133-bf375421a904", "name": "8b6be2ed-087d-4652-a133-bf375421a904", "status": "Failed", "startTime": "2026-08-25T14:40:17.286340435Z", "endTime": "2026-08-25T14:59:19.220463305Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 13:52:37 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc0017db260>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-b4vb4qmlbwh5" to finish creating: GET https://rp.j8970496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6dd6f132-b097-41ed-a77e-dcf30d4582e3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6dd6f132-b097-41ed-a77e-dcf30d4582e3", "name": "6dd6f132-b097-41ed-a77e-dcf30d4582e3", "status": "Failed", "startTime": "2026-08-25T14:42:41.004926697Z", "endTime": "2026-08-25T15:01:49.208965203Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc0017db260>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-b4vb4qmlbwh5" to finish creating: GET https://rp.j8970496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6dd6f132-b097-41ed-a77e-dcf30d4582e3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6dd6f132-b097-41ed-a77e-dcf30d4582e3", "name": "6dd6f132-b097-41ed-a77e-dcf30d4582e3", "status": "Failed", "startTime": "2026-08-25T14:42:41.004926697Z", "endTime": "2026-08-25T15:01:49.208965203Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 13:52:37 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 12 time(s) (4 unknown, 8 known) Firing 1: State: Resolved Started: 2026-08-25T14:51:31Z Ended: 2026-08-25T15:14:29Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8970496-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlmpgvl8u8krpi89rr9e9vj1a2ll6v-b2s3x1u9q6t8r9m", pod="router-54677d69bb-mfrpm", severity="warning" Description: Pod ocm-arohcpci01-2sdlmpgvl8u8krpi89rr9e9vj1a2ll6v-b2s3x1u9q6t8r9m/router-54677d69bb-mfrpm has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-08-25T14:51:31Z Ended: 2026-08-25T15:12:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8970496-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlmkk71i1fsmkh3032328vp5kjd3ik-a4u0o6g1z8e2w2h", pod="router-6c6b797c88-snsbr", severity="warning" Description: Pod ocm-arohcpci01-2sdlmkk71i1fsmkh3032328vp5kjd3ik-a4u0o6g1z8e2w2h/router-6c6b797c88-snsbr has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-08-25T14:51:31Z Ended: 2026-08-25T15:12:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8970496-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlmkk71i1fsmkh3032328vp5kjd3ik-a4u0o6g1z8e2w2h", pod="router-6c6b797c88-ssz4g", severity="warning" Description: Pod ocm-arohcpci01-2sdlmkk71i1fsmkh3032328vp5kjd3ik-a4u0o6g1z8e2w2h/router-6c6b797c88-ssz4g has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-08-25T14:55:28Z Ended: 2026-08-25T15:28:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8970496-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlmbrgto6n0h5o6073npkpf9ttvibr-l4a3s3s1m8o9h0a", pod="router-85db99885d-5dtz8", severity="warning" Description: Pod ocm-arohcpci01-2sdlmbrgto6n0h5o6073npkpf9ttvibr-l4a3s3s1m8o9h0a/router-85db99885d-5dtz8 has been in a non-ready state for longer than 5 minutes.
13:34:16	pull-ci-Azure-ARO-HCP-main-e2e-parallel head ea2dde0	alert, e2e	failed	#5958 (open)	4	e2e (3) ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 13:34:16 UTC · failed at e2e · test SRE should return 409 when boot diagnostics is disabled on a VM · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:433]: failed to create worker nodepool for boot diagnostics test Unexpected error: <fmt.wrapError \| 0xc00019edc0>: failed to create NodePool worker: failed waiting for nodepool="worker" for cluster "sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-prhmmrf56mn9" to finish creating: GET https://rp.j0715520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/90e4482a-a8a6-47af-8a58-d4c088a6709f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/90e4482a-a8a6-47af-8a58-d4c088a6709f", "name": "90e4482a-a8a6-47af-8a58-d4c088a6709f", "status": "Failed", "startTime": "2026-08-25T14:35:42.014090967Z", "endTime": "2026-08-25T14:54:47.709528737Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:433]: failed to create worker nodepool for boot diagnostics test Unexpected error: <fmt.wrapError \| 0xc00019edc0>: failed to create NodePool worker: failed waiting for nodepool="worker" for cluster "sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-prhmmrf56mn9" to finish creating: GET https://rp.j0715520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/90e4482a-a8a6-47af-8a58-d4c088a6709f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/90e4482a-a8a6-47af-8a58-d4c088a6709f", "name": "90e4482a-a8a6-47af-8a58-d4c088a6709f", "status": "Failed", "startTime": "2026-08-25T14:35:42.014090967Z", "endTime": "2026-08-25T14:54:47.709528737Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift 13:34:16 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc00100f360>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-tjq5k8gxthnl" to finish creating: GET https://rp.j0715520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/093b11cd-6fee-48cf-93d0-7cfd46366e65 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/093b11cd-6fee-48cf-93d0-7cfd46366e65", "name": "093b11cd-6fee-48cf-93d0-7cfd46366e65", "status": "Failed", "startTime": "2026-08-25T14:25:06.613580324Z", "endTime": "2026-08-25T14:44:07.687483877Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc00100f360>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-tjq5k8gxthnl" to finish creating: GET https://rp.j0715520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/093b11cd-6fee-48cf-93d0-7cfd46366e65 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/093b11cd-6fee-48cf-93d0-7cfd46366e65", "name": "093b11cd-6fee-48cf-93d0-7cfd46366e65", "status": "Failed", "startTime": "2026-08-25T14:25:06.613580324Z", "endTime": "2026-08-25T14:44:07.687483877Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift 13:34:16 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-6pvw75qcwr4c/cluster-candidate-4-22-42l2l9 should provision Unexpected error: <fmt.wrapError \| 0xc000c7c4c0>: failed to create HCP cluster cluster-candidate-4-22-42l2l9: failed waiting for cluster="cluster-candidate-4-22-42l2l9" in resourcegroup="rg-candidate-4-22-6pvw75qcwr4c" to finish creating: GET https://rp.j0715520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2fbc0167-dc96-49fd-9ffa-f3dfd96dd6ca -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2fbc0167-dc96-49fd-9ffa-f3dfd96dd6ca", "name": "2fbc0167-dc96-49fd-9ffa-f3dfd96dd6ca", "status": "Failed", "startTime": "2026-08-25T14:25:16.547735183Z", "endTime": "2026-08-25T14:44:17.680374337Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-6pvw75qcwr4c/cluster-candidate-4-22-42l2l9 should provision Unexpected error: <fmt.wrapError \| 0xc000c7c4c0>: failed to create HCP cluster cluster-candidate-4-22-42l2l9: failed waiting for cluster="cluster-candidate-4-22-42l2l9" in resourcegroup="rg-candidate-4-22-6pvw75qcwr4c" to finish creating: GET https://rp.j0715520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2fbc0167-dc96-49fd-9ffa-f3dfd96dd6ca -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2fbc0167-dc96-49fd-9ffa-f3dfd96dd6ca", "name": "2fbc0167-dc96-49fd-9ffa-f3dfd96dd6ca", "status": "Failed", "startTime": "2026-08-25T14:25:16.547735183Z", "endTime": "2026-08-25T14:44:17.680374337Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 13:34:16 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 19 time(s) (3 unknown, 16 known) Firing 1: State: Resolved Started: 2026-08-25T14:39:16Z Ended: 2026-08-25T14:45:16Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0715520-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlfhd8vrqvchs6lae3ufqe2gul010f-o9t7t3k4b1c5z3g", pod="cluster-storage-operator-5bb478c9-c6qxc", severity="warning" Description: Pod ocm-arohcpci01-2sdlfhd8vrqvchs6lae3ufqe2gul010f-o9t7t3k4b1c5z3g/cluster-storage-operator-5bb478c9-c6qxc has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-08-25T14:39:16Z Ended: 2026-08-25T14:45:16Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0715520-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlfef9mpp5b2vqt1no14mtaeu69k6k-x0s6g2t5w9k7o6l", pod="cluster-network-operator-9bbf74b9c-m29mf", severity="warning" Description: Pod ocm-arohcpci01-2sdlfef9mpp5b2vqt1no14mtaeu69k6k-x0s6g2t5w9k7o6l/cluster-network-operator-9bbf74b9c-m29mf has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-08-25T14:41:16Z Ended: 2026-08-25T14:47:15Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0715520-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlfgs8aqldhp3lh5r1j2t1paf41f8t-y2s5d6o5w9j5f8c", pod="cluster-network-operator-55cfccd85d-6v4zw", severity="warning" Description: Pod ocm-arohcpci01-2sdlfgs8aqldhp3lh5r1j2t1paf41f8t-y2s5d6o5w9j5f8c/cluster-network-operator-55cfccd85d-6v4zw has been in a non-ready state for longer than 5 minutes.
13:26:03	pull-ci-Azure-ARO-HCP-main-e2e-parallel head a660b6e	alert, e2e	failed	⚠#6660 (open)	15	e2e (14) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 13:26:03 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc0006490a0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-2ppc5kwlqt8f" to finish creating: GET https://rp.j7603200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ba1dde59-7246-414a-86eb-207a9c26ebc0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ba1dde59-7246-414a-86eb-207a9c26ebc0", "name": "ba1dde59-7246-414a-86eb-207a9c26ebc0", "status": "Failed", "startTime": "2026-08-25T14:24:12.940463847Z", "endTime": "2026-08-25T14:43:14.557069884Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc0006490a0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-2ppc5kwlqt8f" to finish creating: GET https://rp.j7603200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ba1dde59-7246-414a-86eb-207a9c26ebc0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ba1dde59-7246-414a-86eb-207a9c26ebc0", "name": "ba1dde59-7246-414a-86eb-207a9c26ebc0", "status": "Failed", "startTime": "2026-08-25T14:24:12.940463847Z", "endTime": "2026-08-25T14:43:14.557069884Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 13:26:03 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc000df4260>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-jtj7cdlldhgj" to finish creating: GET https://rp.j7603200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dc8b29bb-eb97-4d27-a878-9f36e4dc7eb1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dc8b29bb-eb97-4d27-a878-9f36e4dc7eb1", "name": "dc8b29bb-eb97-4d27-a878-9f36e4dc7eb1", "status": "Failed", "startTime": "2026-08-25T14:22:41.424970771Z", "endTime": "2026-08-25T14:41:44.363811133Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc000df4260>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-jtj7cdlldhgj" to finish creating: GET https://rp.j7603200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dc8b29bb-eb97-4d27-a878-9f36e4dc7eb1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dc8b29bb-eb97-4d27-a878-9f36e4dc7eb1", "name": "dc8b29bb-eb97-4d27-a878-9f36e4dc7eb1", "status": "Failed", "startTime": "2026-08-25T14:22:41.424970771Z", "endTime": "2026-08-25T14:41:44.363811133Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 13:26:03 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc000da43a0>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-52lwzkdvrvxq" to finish creating: GET https://rp.j7603200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1c5237ca-aea6-4af4-af59-1bac03bfc222 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1c5237ca-aea6-4af4-af59-1bac03bfc222", "name": "1c5237ca-aea6-4af4-af59-1bac03bfc222", "status": "Failed", "startTime": "2026-08-25T14:23:09.556002909Z", "endTime": "2026-08-25T14:42:14.793443561Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc000da43a0>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-52lwzkdvrvxq" to finish creating: GET https://rp.j7603200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1c5237ca-aea6-4af4-af59-1bac03bfc222 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1c5237ca-aea6-4af4-af59-1bac03bfc222", "name": "1c5237ca-aea6-4af4-af59-1bac03bfc222", "status": "Failed", "startTime": "2026-08-25T14:23:09.556002909Z", "endTime": "2026-08-25T14:42:14.793443561Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 13:26:03 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--vdwvqtzmlhhz/cluster-hs-pre-h4zwsj should provision Unexpected error: <fmt.wrapError \| 0xc000efebc0>: failed to create HCP cluster cluster-hs-pre-h4zwsj: failed waiting for cluster="cluster-hs-pre-h4zwsj" in resourcegroup="rg-hs-pre--vdwvqtzmlhhz" to finish creating: GET https://rp.j7603200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7be718c8-db1c-4834-a67f-c3902ee1577e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7be718c8-db1c-4834-a67f-c3902ee1577e", "name": "7be718c8-db1c-4834-a67f-c3902ee1577e", "status": "Failed", "startTime": "2026-08-25T14:22:26.49157749Z", "endTime": "2026-08-25T14:41:34.372350557Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--vdwvqtzmlhhz/cluster-hs-pre-h4zwsj should provision Unexpected error: <fmt.wrapError \| 0xc000efebc0>: failed to create HCP cluster cluster-hs-pre-h4zwsj: failed waiting for cluster="cluster-hs-pre-h4zwsj" in resourcegroup="rg-hs-pre--vdwvqtzmlhhz" to finish creating: GET https://rp.j7603200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7be718c8-db1c-4834-a67f-c3902ee1577e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7be718c8-db1c-4834-a67f-c3902ee1577e", "name": "7be718c8-db1c-4834-a67f-c3902ee1577e", "status": "Failed", "startTime": "2026-08-25T14:22:26.49157749Z", "endTime": "2026-08-25T14:41:34.372350557Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 13:26:03 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc000f9e160>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-xdpzglqjl7k5" to finish creating: GET https://rp.j7603200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/22657958-a1fd-4572-9f54-a3f4670af3f8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/22657958-a1fd-4572-9f54-a3f4670af3f8", "name": "22657958-a1fd-4572-9f54-a3f4670af3f8", "status": "Failed", "startTime": "2026-08-25T14:25:10.9485512Z", "endTime": "2026-08-25T14:44:14.603331408Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc000f9e160>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-xdpzglqjl7k5" to finish creating: GET https://rp.j7603200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/22657958-a1fd-4572-9f54-a3f4670af3f8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/22657958-a1fd-4572-9f54-a3f4670af3f8", "name": "22657958-a1fd-4572-9f54-a3f4670af3f8", "status": "Failed", "startTime": "2026-08-25T14:25:10.9485512Z", "endTime": "2026-08-25T14:44:14.603331408Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 13:26:03 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-jprm6wpttrf7/cluster-candidate-5-0-n79n6k should provision Unexpected error: <fmt.wrapError \| 0xc000f5a120>: failed to create HCP cluster cluster-candidate-5-0-n79n6k: failed waiting for cluster="cluster-candidate-5-0-n79n6k" in resourcegroup="rg-candidate-5-0-jprm6wpttrf7" to finish creating: GET https://rp.j7603200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b5c105bf-67e2-4a17-aebe-38cc80b7408e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b5c105bf-67e2-4a17-aebe-38cc80b7408e", "name": "b5c105bf-67e2-4a17-aebe-38cc80b7408e", "status": "Failed", "startTime": "2026-08-25T14:22:22.605993872Z", "endTime": "2026-08-25T14:41:24.396649573Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-jprm6wpttrf7/cluster-candidate-5-0-n79n6k should provision Unexpected error: <fmt.wrapError \| 0xc000f5a120>: failed to create HCP cluster cluster-candidate-5-0-n79n6k: failed waiting for cluster="cluster-candidate-5-0-n79n6k" in resourcegroup="rg-candidate-5-0-jprm6wpttrf7" to finish creating: GET https://rp.j7603200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b5c105bf-67e2-4a17-aebe-38cc80b7408e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b5c105bf-67e2-4a17-aebe-38cc80b7408e", "name": "b5c105bf-67e2-4a17-aebe-38cc80b7408e", "status": "Failed", "startTime": "2026-08-25T14:22:22.605993872Z", "endTime": "2026-08-25T14:41:24.396649573Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 13:26:03 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-2n6qpzrjsvhw/cluster-candidate-4-21-zb4zw7 should provision Unexpected error: <fmt.wrapError \| 0xc000e7e600>: failed to create HCP cluster cluster-candidate-4-21-zb4zw7: failed waiting for cluster="cluster-candidate-4-21-zb4zw7" in resourcegroup="rg-candidate-4-21-2n6qpzrjsvhw" to finish creating: GET https://rp.j7603200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5f39df5e-409a-4851-b372-be5717487b61 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5f39df5e-409a-4851-b372-be5717487b61", "name": "5f39df5e-409a-4851-b372-be5717487b61", "status": "Failed", "startTime": "2026-08-25T14:22:26.840054334Z", "endTime": "2026-08-25T14:41:34.377011529Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-2n6qpzrjsvhw/cluster-candidate-4-21-zb4zw7 should provision Unexpected error: <fmt.wrapError \| 0xc000e7e600>: failed to create HCP cluster cluster-candidate-4-21-zb4zw7: failed waiting for cluster="cluster-candidate-4-21-zb4zw7" in resourcegroup="rg-candidate-4-21-2n6qpzrjsvhw" to finish creating: GET https://rp.j7603200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5f39df5e-409a-4851-b372-be5717487b61 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5f39df5e-409a-4851-b372-be5717487b61", "name": "5f39df5e-409a-4851-b372-be5717487b61", "status": "Failed", "startTime": "2026-08-25T14:22:26.840054334Z", "endTime": "2026-08-25T14:41:34.377011529Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 13:26:03 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:739]: failed to create nodepool npdg-422-420 Unexpected error: <fmt.wrapError \| 0xc0004c9840>: failed to create NodePool npdg-422-420: failed waiting for nodepool="npdg-422-420" for cluster "np-dg-minor-wpzzs6" in resourcegroup="rg-np-dg-minor-wpzzs6-6jql8zwjxxlb" to finish creating: GET https://rp.j7603200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/73b2139d-e941-4022-8bf3-72abae50ef1a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/73b2139d-e941-4022-8bf3-72abae50ef1a", "name": "73b2139d-e941-4022-8bf3-72abae50ef1a", "status": "Failed", "startTime": "2026-08-25T14:35:50.588975119Z", "endTime": "2026-08-25T14:54:53.875483949Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:739]: failed to create nodepool npdg-422-420 Unexpected error: <fmt.wrapError \| 0xc0004c9840>: failed to create NodePool npdg-422-420: failed waiting for nodepool="npdg-422-420" for cluster "np-dg-minor-wpzzs6" in resourcegroup="rg-np-dg-minor-wpzzs6-6jql8zwjxxlb" to finish creating: GET https://rp.j7603200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/73b2139d-e941-4022-8bf3-72abae50ef1a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/73b2139d-e941-4022-8bf3-72abae50ef1a", "name": "73b2139d-e941-4022-8bf3-72abae50ef1a", "status": "Failed", "startTime": "2026-08-25T14:35:50.588975119Z", "endTime": "2026-08-25T14:54:53.875483949Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 13:26:03 UTC · failed at e2e · test SRE can pause schedules to stop backup execution for an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000e0ef80>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-bhfbpkdqqdrn" to finish creating: GET https://rp.j7603200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c0dda70b-836c-4894-a3d0-be51c86b05cf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c0dda70b-836c-4894-a3d0-be51c86b05cf", "name": "c0dda70b-836c-4894-a3d0-be51c86b05cf", "status": "Failed", "startTime": "2026-08-25T14:22:44.500975Z", "endTime": "2026-08-25T14:41:44.271185619Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000e0ef80>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-bhfbpkdqqdrn" to finish creating: GET https://rp.j7603200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c0dda70b-836c-4894-a3d0-be51c86b05cf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c0dda70b-836c-4894-a3d0-be51c86b05cf", "name": "c0dda70b-836c-4894-a3d0-be51c86b05cf", "status": "Failed", "startTime": "2026-08-25T14:22:44.500975Z", "endTime": "2026-08-25T14:41:44.271185619Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator]; provider Microsoft.RedHatOpenShift 13:26:03 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-4z7lpz" with version 5.0.0-ec.5 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000e04260>: failed to create HCP cluster cluster-zstream-5-0-4z7lpz: failed waiting for cluster="cluster-zstream-5-0-4z7lpz" in resourcegroup="rg-zstream-upgrade-5-0-4tgw65s225n8" to finish creating: GET https://rp.j7603200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8e8c47b5-1463-40b5-871f-63030d9c0472 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8e8c47b5-1463-40b5-871f-63030d9c0472", "name": "8e8c47b5-1463-40b5-871f-63030d9c0472", "status": "Failed", "startTime": "2026-08-25T14:22:33.195161833Z", "endTime": "2026-08-25T14:41:34.370521419Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-4z7lpz" with version 5.0.0-ec.5 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000e04260>: failed to create HCP cluster cluster-zstream-5-0-4z7lpz: failed waiting for cluster="cluster-zstream-5-0-4z7lpz" in resourcegroup="rg-zstream-upgrade-5-0-4tgw65s225n8" to finish creating: GET https://rp.j7603200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8e8c47b5-1463-40b5-871f-63030d9c0472 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8e8c47b5-1463-40b5-871f-63030d9c0472", "name": "8e8c47b5-1463-40b5-871f-63030d9c0472", "status": "Failed", "startTime": "2026-08-25T14:22:33.195161833Z", "endTime": "2026-08-25T14:41:34.370521419Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 13:26:03 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-tzvdrp4rlnjq/cluster-candidate-4-20-txnzsg should provision Unexpected error: <fmt.wrapError \| 0xc000f05420>: failed to create HCP cluster cluster-candidate-4-20-txnzsg: failed waiting for cluster="cluster-candidate-4-20-txnzsg" in resourcegroup="rg-candidate-4-20-tzvdrp4rlnjq" to finish creating: GET https://rp.j7603200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bb25a348-e3b8-4c82-9665-cf01bc14f70f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bb25a348-e3b8-4c82-9665-cf01bc14f70f", "name": "bb25a348-e3b8-4c82-9665-cf01bc14f70f", "status": "Failed", "startTime": "2026-08-25T14:22:26.331653993Z", "endTime": "2026-08-25T14:41:34.332024135Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-tzvdrp4rlnjq/cluster-candidate-4-20-txnzsg should provision Unexpected error: <fmt.wrapError \| 0xc000f05420>: failed to create HCP cluster cluster-candidate-4-20-txnzsg: failed waiting for cluster="cluster-candidate-4-20-txnzsg" in resourcegroup="rg-candidate-4-20-tzvdrp4rlnjq" to finish creating: GET https://rp.j7603200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bb25a348-e3b8-4c82-9665-cf01bc14f70f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bb25a348-e3b8-4c82-9665-cf01bc14f70f", "name": "bb25a348-e3b8-4c82-9665-cf01bc14f70f", "status": "Failed", "startTime": "2026-08-25T14:22:26.331653993Z", "endTime": "2026-08-25T14:41:34.332024135Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator, ignition-server-proxy, kube-controller-manager, router]; provider Microsoft.RedHatOpenShift 13:26:03 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc000b0afc0>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-gvltc2gr7wfl" to finish creating: GET https://rp.j7603200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9a8b4b1e-639e-425c-8c76-d099661899e0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9a8b4b1e-639e-425c-8c76-d099661899e0", "name": "9a8b4b1e-639e-425c-8c76-d099661899e0", "status": "Failed", "startTime": "2026-08-25T14:24:12.827625738Z", "endTime": "2026-08-25T14:43:14.39497302Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 2 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc000b0afc0>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-gvltc2gr7wfl" to finish creating: GET https://rp.j7603200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9a8b4b1e-639e-425c-8c76-d099661899e0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9a8b4b1e-639e-425c-8c76-d099661899e0", "name": "9a8b4b1e-639e-425c-8c76-d099661899e0", "status": "Failed", "startTime": "2026-08-25T14:24:12.827625738Z", "endTime": "2026-08-25T14:43:14.39497302Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 2 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred error for empty VM name should contain 400 status code 13:26:03 UTC · failed at e2e · test SRE should be able to retrieve serial console logs for a VM · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:359]: error for empty VM name should contain 400 status code Expected <string>: ... to contain substring <string>: ... fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:359]: error for empty VM name should contain 400 status code Expected <string>: ... to contain substring <string>: ... ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 13:26:03 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc00196e760>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-dsvpvl48ct65" to finish creating: GET https://rp.j7603200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aaa93704-3b70-47e4-b524-6a5f42adba5d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aaa93704-3b70-47e4-b524-6a5f42adba5d", "name": "aaa93704-3b70-47e4-b524-6a5f42adba5d", "status": "Failed", "startTime": "2026-08-25T14:22:38.164192939Z", "endTime": "2026-08-25T14:41:44.371074479Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc00196e760>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-dsvpvl48ct65" to finish creating: GET https://rp.j7603200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aaa93704-3b70-47e4-b524-6a5f42adba5d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aaa93704-3b70-47e4-b524-6a5f42adba5d", "name": "aaa93704-3b70-47e4-b524-6a5f42adba5d", "status": "Failed", "startTime": "2026-08-25T14:22:38.164192939Z", "endTime": "2026-08-25T14:41:44.371074479Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 13:26:03 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 11 time(s) (9 unknown, 2 known) Firing 1: State: Resolved Started: 2026-08-25T14:35:07Z Ended: 2026-08-25T14:59:09Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7603200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlejnfp7395pe7ttfrm2o3t3i5i0s1-private-kas", pod="router-7666b5d5c4-zznx6", severity="warning" Description: Pod ocm-arohcpci01-2sdlejnfp7395pe7ttfrm2o3t3i5i0s1-private-kas/router-7666b5d5c4-zznx6 has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-08-25T14:35:07Z Ended: 2026-08-25T14:52:08Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7603200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlejnfp7395pe7ttfrm2o3t3i5i0s1-private-kas", pod="router-7666b5d5c4-kfsdq", severity="warning" Description: Pod ocm-arohcpci01-2sdlejnfp7395pe7ttfrm2o3t3i5i0s1-private-kas/router-7666b5d5c4-kfsdq has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-08-25T14:35:07Z Ended: 2026-08-25T14:53:08Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7603200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdle8r3r5ancl77mi9gtmfo4opktruc-s1p8l7t8j3h1w3u", pod="router-d94cfcdbd-4nj4q", severity="warning" Description: Pod ocm-arohcpci01-2sdle8r3r5ancl77mi9gtmfo4opktruc-s1p8l7t8j3h1w3u/router-d94cfcdbd-4nj4q has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-08-25T14:36:07Z Ended: 2026-08-25T14:52:08Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7603200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdldvge2kpq5odr57p465kue54uh9fi-a9a5z8y1j0x9z6j", pod="router-657fb7bdcf-tc592", severity="warning" Description: Pod ocm-arohcpci01-2sdldvge2kpq5odr57p465kue54uh9fi-a9a5z8y1j0x9z6j/router-657fb7bdcf-tc592 has been in a non-ready state for longer than 5 minutes. Firing 5: State: Resolved Started: 2026-08-25T14:36:08Z Ended: 2026-08-25T14:52:08Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7603200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdle8svkin145smgvdv0pdgj1d7mcgh-p5b7o7v3h7a1b0f", pod="router-5bc7d4c559-rpll4", severity="warning" Description: Pod ocm-arohcpci01-2sdle8svkin145smgvdv0pdgj1d7mcgh-p5b7o7v3h7a1b0f/router-5bc7d4c559-rpll4 has been in a non-ready state for longer than 5 minutes. Firing 6: State: Resolved Started: 2026-08-25T14:37:07Z Ended: 2026-08-25T14:43:07Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7603200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlf3jn7opj4balf1jd2v7mg84482e1-cilium-cluster", pod="router-648dc7cb9b-8mv4d", severity="warning" Description: Pod ocm-arohcpci01-2sdlf3jn7opj4balf1jd2v7mg84482e1-cilium-cluster/router-648dc7cb9b-8mv4d has been in a non-ready state for longer than 5 minutes. Firing 7: State: Resolved Started: 2026-08-25T14:37:07Z Ended: 2026-08-25T14:53:08Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7603200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlebjb4f830k7dqj7mi38s5u8unig7-l1h8m2e0e1k3j3a", pod="router-756bd757b9-sxttr", severity="warning" Description: Pod ocm-arohcpci01-2sdlebjb4f830k7dqj7mi38s5u8unig7-l1h8m2e0e1k3j3a/router-756bd757b9-sxttr has been in a non-ready state for longer than 5 minutes. Firing 8: State: Resolved Started: 2026-08-25T14:37:07Z Ended: 2026-08-25T14:58:06Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7603200-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdle7lje546dmg1m8eo3r2g0l88kh33-p4w5p2d5h6o0y9w", pod="router-6df7459d5-h8fkt", severity="warning" Description: Pod ocm-arohcpci01-2sdle7lje546dmg1m8eo3r2g0l88kh33-p4w5p2d5h6o0y9w/router-6df7459d5-h8fkt has been in a non-ready state for longer than 5 minutes. Firing 9: State: Resolved Started: 2026-08-25T14:37:08Z Ended: 2026-08-25T14:55:07Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7603200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlebjb4f830k7dqj7mi38s5u8unig7-l1h8m2e0e1k3j3a", pod="router-756bd757b9-pmmhv", severity="warning" Description: Pod ocm-arohcpci01-2sdlebjb4f830k7dqj7mi38s5u8unig7-l1h8m2e0e1k3j3a/router-756bd757b9-pmmhv has been in a non-ready state for longer than 5 minutes.
13:21:13	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c9b8452	e2e	failed	⚠#6584 (open)	12	e2e (12) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 13:21:13 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260630preview and verify the ingress is internal · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc000ee4940>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-h62zblbg9qvp" to finish creating: GET https://rp.j7004544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/708b2a4f-d727-42ba-851b-4c217c4509be -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/708b2a4f-d727-42ba-851b-4c217c4509be", "name": "708b2a4f-d727-42ba-851b-4c217c4509be", "status": "Failed", "startTime": "2026-08-25T14:21:09.885838713Z", "endTime": "2026-08-25T14:40:13.678306778Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc000ee4940>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-h62zblbg9qvp" to finish creating: GET https://rp.j7004544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/708b2a4f-d727-42ba-851b-4c217c4509be -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/708b2a4f-d727-42ba-851b-4c217c4509be", "name": "708b2a4f-d727-42ba-851b-4c217c4509be", "status": "Failed", "startTime": "2026-08-25T14:21:09.885838713Z", "endTime": "2026-08-25T14:40:13.678306778Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator, ingress-operator]; provider Microsoft.RedHatOpenShift 13:21:13 UTC · failed at e2e · test SRE can pause schedules to stop backup execution for an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc0019daa40>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-g5pwkrb85fq6" to finish creating: GET https://rp.j7004544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a089fdb6-b47d-4203-b58e-db6f9734d598 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a089fdb6-b47d-4203-b58e-db6f9734d598", "name": "a089fdb6-b47d-4203-b58e-db6f9734d598", "status": "Failed", "startTime": "2026-08-25T14:21:12.44361008Z", "endTime": "2026-08-25T14:40:13.632066828Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc0019daa40>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-g5pwkrb85fq6" to finish creating: GET https://rp.j7004544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a089fdb6-b47d-4203-b58e-db6f9734d598 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a089fdb6-b47d-4203-b58e-db6f9734d598", "name": "a089fdb6-b47d-4203-b58e-db6f9734d598", "status": "Failed", "startTime": "2026-08-25T14:21:12.44361008Z", "endTime": "2026-08-25T14:40:13.632066828Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 13:21:13 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc0019ee900>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-bczkkzw624lz" to finish creating: GET https://rp.j7004544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0b1cb529-87ca-48d5-8037-5361bfdad717 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0b1cb529-87ca-48d5-8037-5361bfdad717", "name": "0b1cb529-87ca-48d5-8037-5361bfdad717", "status": "Failed", "startTime": "2026-08-25T14:21:32.618109568Z", "endTime": "2026-08-25T14:40:33.581291257Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, openshift-route-controller-manager, cluster-node-tuning-operator, cluster-network-operator, catalog-operator, olm-operator, ingress-operator, redhat-operators-catalog, packageserver, community-operators-catalog, hosted-cluster-config-operator, cluster-storage-operator, openshift-controller-manager, certified-operators-catalog, cluster-version-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 2 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc0019ee900>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-bczkkzw624lz" to finish creating: GET https://rp.j7004544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0b1cb529-87ca-48d5-8037-5361bfdad717 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0b1cb529-87ca-48d5-8037-5361bfdad717", "name": "0b1cb529-87ca-48d5-8037-5361bfdad717", "status": "Failed", "startTime": "2026-08-25T14:21:32.618109568Z", "endTime": "2026-08-25T14:40:33.581291257Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, openshift-route-controller-manager, cluster-node-tuning-operator, cluster-network-operator, catalog-operator, olm-operator, ingress-operator, redhat-operators-catalog, packageserver, community-operators-catalog, hosted-cluster-config-operator, cluster-storage-operator, openshift-controller-manager, certified-operators-catalog, cluster-version-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 2 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator]; provider Microsoft.RedHatOpenShift 13:21:13 UTC · failed at e2e · test SRE should be able to retrieve serial console logs for a VM · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:308]: failed to create HCP cluster "sre-hcp-cluster-sc" for serial console test Unexpected error: <fmt.wrapError \| 0xc00116a220>: failed to create HCP cluster sre-hcp-cluster-sc: failed waiting for cluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-b9gsdkcb5wjn" to finish creating: GET https://rp.j7004544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ac2c2af3-451a-4ece-96a8-63d551193d0d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ac2c2af3-451a-4ece-96a8-63d551193d0d", "name": "ac2c2af3-451a-4ece-96a8-63d551193d0d", "status": "Failed", "startTime": "2026-08-25T14:21:09.74317293Z", "endTime": "2026-08-25T14:40:13.66125404Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:308]: failed to create HCP cluster "sre-hcp-cluster-sc" for serial console test Unexpected error: <fmt.wrapError \| 0xc00116a220>: failed to create HCP cluster sre-hcp-cluster-sc: failed waiting for cluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-b9gsdkcb5wjn" to finish creating: GET https://rp.j7004544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ac2c2af3-451a-4ece-96a8-63d551193d0d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ac2c2af3-451a-4ece-96a8-63d551193d0d", "name": "ac2c2af3-451a-4ece-96a8-63d551193d0d", "status": "Failed", "startTime": "2026-08-25T14:21:09.74317293Z", "endTime": "2026-08-25T14:40:13.66125404Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 13:21:13 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:460]: failed to create nodepool nps-420-422 at version 4.20.35 Unexpected error: <fmt.wrapError \| 0xc000e09e40>: failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-trdpwk" in resourcegroup="rg-np-skip-minor-trdpwk-wpbsh8r8pf85" to finish creating: GET https://rp.j7004544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4046da41-b84c-482b-b6f6-53a4043531fd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4046da41-b84c-482b-b6f6-53a4043531fd", "name": "4046da41-b84c-482b-b6f6-53a4043531fd", "status": "Failed", "startTime": "2026-08-25T14:35:48.059407214Z", "endTime": "2026-08-25T14:54:53.654727258Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:460]: failed to create nodepool nps-420-422 at version 4.20.35 Unexpected error: <fmt.wrapError \| 0xc000e09e40>: failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-trdpwk" in resourcegroup="rg-np-skip-minor-trdpwk-wpbsh8r8pf85" to finish creating: GET https://rp.j7004544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4046da41-b84c-482b-b6f6-53a4043531fd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4046da41-b84c-482b-b6f6-53a4043531fd", "name": "4046da41-b84c-482b-b6f6-53a4043531fd", "status": "Failed", "startTime": "2026-08-25T14:35:48.059407214Z", "endTime": "2026-08-25T14:54:53.654727258Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator, dns-operator]; provider Microsoft.RedHatOpenShift 13:21:13 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:721]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc0001a5060>: failed to create HCP cluster np-dg-minor-gzvdzt: failed waiting for cluster="np-dg-minor-gzvdzt" in resourcegroup="rg-np-dg-minor-gzvdzt-zrllgdp7qt99" to finish creating: GET https://rp.j7004544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5b036348-ef9c-491d-be11-832d77741098 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5b036348-ef9c-491d-be11-832d77741098", "name": "5b036348-ef9c-491d-be11-832d77741098", "status": "Failed", "startTime": "2026-08-25T14:21:00.551807818Z", "endTime": "2026-08-25T14:40:03.659175976Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:721]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc0001a5060>: failed to create HCP cluster np-dg-minor-gzvdzt: failed waiting for cluster="np-dg-minor-gzvdzt" in resourcegroup="rg-np-dg-minor-gzvdzt-zrllgdp7qt99" to finish creating: GET https://rp.j7004544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5b036348-ef9c-491d-be11-832d77741098 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5b036348-ef9c-491d-be11-832d77741098", "name": "5b036348-ef9c-491d-be11-832d77741098", "status": "Failed", "startTime": "2026-08-25T14:21:00.551807818Z", "endTime": "2026-08-25T14:40:03.659175976Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 13:21:13 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-2znfmbcxp8tg/cluster-candidate-4-22-kbnpqt should provision Unexpected error: <fmt.wrapError \| 0xc000ff0200>: failed to create HCP cluster cluster-candidate-4-22-kbnpqt: failed waiting for cluster="cluster-candidate-4-22-kbnpqt" in resourcegroup="rg-candidate-4-22-2znfmbcxp8tg" to finish creating: GET https://rp.j7004544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f24431ca-76c6-43bb-b4ca-aaa1fba71d4b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f24431ca-76c6-43bb-b4ca-aaa1fba71d4b", "name": "f24431ca-76c6-43bb-b4ca-aaa1fba71d4b", "status": "Failed", "startTime": "2026-08-25T14:21:12.456614245Z", "endTime": "2026-08-25T14:40:13.674948976Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-2znfmbcxp8tg/cluster-candidate-4-22-kbnpqt should provision Unexpected error: <fmt.wrapError \| 0xc000ff0200>: failed to create HCP cluster cluster-candidate-4-22-kbnpqt: failed waiting for cluster="cluster-candidate-4-22-kbnpqt" in resourcegroup="rg-candidate-4-22-2znfmbcxp8tg" to finish creating: GET https://rp.j7004544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f24431ca-76c6-43bb-b4ca-aaa1fba71d4b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f24431ca-76c6-43bb-b4ca-aaa1fba71d4b", "name": "f24431ca-76c6-43bb-b4ca-aaa1fba71d4b", "status": "Failed", "startTime": "2026-08-25T14:21:12.456614245Z", "endTime": "2026-08-25T14:40:13.674948976Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 13:21:13 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--v8ffjmksbzdl/cluster-hs-pre-8jd46w should provision Unexpected error: <fmt.wrapError \| 0xc001938ac0>: failed to create HCP cluster cluster-hs-pre-8jd46w: failed waiting for cluster="cluster-hs-pre-8jd46w" in resourcegroup="rg-hs-pre--v8ffjmksbzdl" to finish creating: GET https://rp.j7004544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/267a7d0a-658c-4265-8e01-7307d8e5f38b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/267a7d0a-658c-4265-8e01-7307d8e5f38b", "name": "267a7d0a-658c-4265-8e01-7307d8e5f38b", "status": "Failed", "startTime": "2026-08-25T14:20:39.09953391Z", "endTime": "2026-08-25T14:39:43.525617914Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--v8ffjmksbzdl/cluster-hs-pre-8jd46w should provision Unexpected error: <fmt.wrapError \| 0xc001938ac0>: failed to create HCP cluster cluster-hs-pre-8jd46w: failed waiting for cluster="cluster-hs-pre-8jd46w" in resourcegroup="rg-hs-pre--v8ffjmksbzdl" to finish creating: GET https://rp.j7004544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/267a7d0a-658c-4265-8e01-7307d8e5f38b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/267a7d0a-658c-4265-8e01-7307d8e5f38b", "name": "267a7d0a-658c-4265-8e01-7307d8e5f38b", "status": "Failed", "startTime": "2026-08-25T14:20:39.09953391Z", "endTime": "2026-08-25T14:39:43.525617914Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 13:21:13 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:137]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster" Unexpected error: <fmt.wrapError \| 0xc000da2d80>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-x6vhg5f425wn" to finish creating: GET https://rp.j7004544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/85b408c1-3b72-4317-9f37-da38ef217efb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/85b408c1-3b72-4317-9f37-da38ef217efb", "name": "85b408c1-3b72-4317-9f37-da38ef217efb", "status": "Failed", "startTime": "2026-08-25T14:37:38.068129685Z", "endTime": "2026-08-25T14:56:43.524475606Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:137]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster" Unexpected error: <fmt.wrapError \| 0xc000da2d80>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-x6vhg5f425wn" to finish creating: GET https://rp.j7004544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/85b408c1-3b72-4317-9f37-da38ef217efb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/85b408c1-3b72-4317-9f37-da38ef217efb", "name": "85b408c1-3b72-4317-9f37-da38ef217efb", "status": "Failed", "startTime": "2026-08-25T14:37:38.068129685Z", "endTime": "2026-08-25T14:56:43.524475606Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager, catalog-operator, hosted-cluster-config-operator, machine-approver, olm-operator, openshift-apiserver, packageserver, router]; provider Microsoft.RedHatOpenShift 13:21:13 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc002200620>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-2pmrbkkbtqsr" to finish creating: GET https://rp.j7004544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/27a59a4e-4c36-4607-a1b2-03e8280030f6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/27a59a4e-4c36-4607-a1b2-03e8280030f6", "name": "27a59a4e-4c36-4607-a1b2-03e8280030f6", "status": "Failed", "startTime": "2026-08-25T14:23:13.29802062Z", "endTime": "2026-08-25T14:42:13.632929999Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: azure-cloud-controller-manager, dns-operator, cluster-version-operator, cluster-storage-operator, certified-operators-catalog, machine-approver, openshift-controller-manager, catalog-operator, csi-snapshot-controller-operator, redhat-marketplace-catalog, redhat-operators-catalog, openshift-apiserver, openshift-route-controller-manager, cluster-network-operator, cluster-node-tuning-operator, olm-operator, community-operators-catalog, cluster-policy-controller, hosted-cluster-config-operator, ingress-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 3 unavailable replicas, packageserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc002200620>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-2pmrbkkbtqsr" to finish creating: GET https://rp.j7004544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/27a59a4e-4c36-4607-a1b2-03e8280030f6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/27a59a4e-4c36-4607-a1b2-03e8280030f6", "name": "27a59a4e-4c36-4607-a1b2-03e8280030f6", "status": "Failed", "startTime": "2026-08-25T14:23:13.29802062Z", "endTime": "2026-08-25T14:42:13.632929999Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: azure-cloud-controller-manager, dns-operator, cluster-version-operator, cluster-storage-operator, certified-operators-catalog, machine-approver, openshift-controller-manager, catalog-operator, csi-snapshot-controller-operator, redhat-marketplace-catalog, redhat-operators-catalog, openshift-apiserver, openshift-route-controller-manager, cluster-network-operator, cluster-node-tuning-operator, olm-operator, community-operators-catalog, cluster-policy-controller, hosted-cluster-config-operator, ingress-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 3 unavailable replicas, packageserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator, ignition-server-proxy, kube-controller-manager, router]; provider Microsoft.RedHatOpenShift 13:21:13 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc0011e26e0>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-lf47q9x8cxz7" to finish creating: GET https://rp.j7004544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/812a5a7d-5729-4953-b916-1cbe8aba438f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/812a5a7d-5729-4953-b916-1cbe8aba438f", "name": "812a5a7d-5729-4953-b916-1cbe8aba438f", "status": "Failed", "startTime": "2026-08-25T14:22:21.177808774Z", "endTime": "2026-08-25T14:41:23.66756422Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc0011e26e0>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-lf47q9x8cxz7" to finish creating: GET https://rp.j7004544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/812a5a7d-5729-4953-b916-1cbe8aba438f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/812a5a7d-5729-4953-b916-1cbe8aba438f", "name": "812a5a7d-5729-4953-b916-1cbe8aba438f", "status": "Failed", "startTime": "2026-08-25T14:22:21.177808774Z", "endTime": "2026-08-25T14:41:23.66756422Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator, cluster-network-operator, cluster-storage-operator, csi-snapshot-controller-operator, dns-operator, ingress-operator, packageserver]; provider Microsoft.RedHatOpenShift 13:21:13 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-fl7ppg8w76k6/cluster-candidate-5-0-9h5fbf should provision Unexpected error: <fmt.wrapError \| 0xc0013b0300>: failed to create HCP cluster cluster-candidate-5-0-9h5fbf: failed waiting for cluster="cluster-candidate-5-0-9h5fbf" in resourcegroup="rg-candidate-5-0-fl7ppg8w76k6" to finish creating: GET https://rp.j7004544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3e7b99ad-c6a9-48e9-9d4c-e880771fde30 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3e7b99ad-c6a9-48e9-9d4c-e880771fde30", "name": "3e7b99ad-c6a9-48e9-9d4c-e880771fde30", "status": "Failed", "startTime": "2026-08-25T14:20:49.919147854Z", "endTime": "2026-08-25T14:39:53.656202045Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, csi-snapshot-controller-operator, packageserver, cluster-storage-operator, dns-operator, ingress-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, packageserver deployment has 2 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-fl7ppg8w76k6/cluster-candidate-5-0-9h5fbf should provision Unexpected error: <fmt.wrapError \| 0xc0013b0300>: failed to create HCP cluster cluster-candidate-5-0-9h5fbf: failed waiting for cluster="cluster-candidate-5-0-9h5fbf" in resourcegroup="rg-candidate-5-0-fl7ppg8w76k6" to finish creating: GET https://rp.j7004544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3e7b99ad-c6a9-48e9-9d4c-e880771fde30 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3e7b99ad-c6a9-48e9-9d4c-e880771fde30", "name": "3e7b99ad-c6a9-48e9-9d4c-e880771fde30", "status": "Failed", "startTime": "2026-08-25T14:20:49.919147854Z", "endTime": "2026-08-25T14:39:53.656202045Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, csi-snapshot-controller-operator, packageserver, cluster-storage-operator, dns-operator, ingress-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, packageserver deployment has 2 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred
13:20:40	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 2a2bf6c	alert, e2e	failed	⚠#6641 (open)	23	e2e (21) ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 13:20:40 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260630preview and verify the ingress is internal · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc000a3c8e0>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-4z5qknlhfz6j" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00c871f1-c55a-462f-87ca-c17ffa7cfa45 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00c871f1-c55a-462f-87ca-c17ffa7cfa45", "name": "00c871f1-c55a-462f-87ca-c17ffa7cfa45", "status": "Failed", "startTime": "2026-08-25T14:23:29.653354515Z", "endTime": "2026-08-25T14:42:33.765546046Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc000a3c8e0>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-4z5qknlhfz6j" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00c871f1-c55a-462f-87ca-c17ffa7cfa45 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00c871f1-c55a-462f-87ca-c17ffa7cfa45", "name": "00c871f1-c55a-462f-87ca-c17ffa7cfa45", "status": "Failed", "startTime": "2026-08-25T14:23:29.653354515Z", "endTime": "2026-08-25T14:42:33.765546046Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 13:20:40 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc000e099e0>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-wdfkhlwcpc48" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3126ea30-d507-4147-af14-e885160a5806 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3126ea30-d507-4147-af14-e885160a5806", "name": "3126ea30-d507-4147-af14-e885160a5806", "status": "Failed", "startTime": "2026-08-25T14:24:12.35552824Z", "endTime": "2026-08-25T14:43:13.692702059Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, olm-operator, redhat-operators-catalog, packageserver, cluster-storage-operator, cluster-image-registry-operator, hosted-cluster-config-operator, ingress-operator, community-operators-catalog, dns-operator, csi-snapshot-controller-operator, certified-operators-catalog, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc000e099e0>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-wdfkhlwcpc48" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3126ea30-d507-4147-af14-e885160a5806 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3126ea30-d507-4147-af14-e885160a5806", "name": "3126ea30-d507-4147-af14-e885160a5806", "status": "Failed", "startTime": "2026-08-25T14:24:12.35552824Z", "endTime": "2026-08-25T14:43:13.692702059Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, olm-operator, redhat-operators-catalog, packageserver, cluster-storage-operator, cluster-image-registry-operator, hosted-cluster-config-operator, ingress-operator, community-operators-catalog, dns-operator, csi-snapshot-controller-operator, certified-operators-catalog, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 13:20:40 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-28m5d4v9fkbw/cluster-candidate-4-22-ng5jqz should provision Unexpected error: <fmt.wrapError \| 0xc000c8a8a0>: failed to create HCP cluster cluster-candidate-4-22-ng5jqz: failed waiting for cluster="cluster-candidate-4-22-ng5jqz" in resourcegroup="rg-candidate-4-22-28m5d4v9fkbw" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9bd2042e-01bd-41c0-98fb-a47ff3536356 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9bd2042e-01bd-41c0-98fb-a47ff3536356", "name": "9bd2042e-01bd-41c0-98fb-a47ff3536356", "status": "Failed", "startTime": "2026-08-25T14:22:58.928780883Z", "endTime": "2026-08-25T14:42:03.744023943Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-28m5d4v9fkbw/cluster-candidate-4-22-ng5jqz should provision Unexpected error: <fmt.wrapError \| 0xc000c8a8a0>: failed to create HCP cluster cluster-candidate-4-22-ng5jqz: failed waiting for cluster="cluster-candidate-4-22-ng5jqz" in resourcegroup="rg-candidate-4-22-28m5d4v9fkbw" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9bd2042e-01bd-41c0-98fb-a47ff3536356 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9bd2042e-01bd-41c0-98fb-a47ff3536356", "name": "9bd2042e-01bd-41c0-98fb-a47ff3536356", "status": "Failed", "startTime": "2026-08-25T14:22:58.928780883Z", "endTime": "2026-08-25T14:42:03.744023943Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-controller-manager]; provider Microsoft.RedHatOpenShift 13:20:40 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0xc000e88e20>: failed to create HCP cluster ea-listhfgh7b: failed waiting for cluster="ea-listhfgh7b" in resourcegroup="ea-list-6p4sr9k5q7bf" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/62436e44-f35d-4583-a471-e3b8c1476144 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/62436e44-f35d-4583-a471-e3b8c1476144", "name": "62436e44-f35d-4583-a471-e3b8c1476144", "status": "Failed", "startTime": "2026-08-25T14:23:30.392692104Z", "endTime": "2026-08-25T14:42:33.727154801Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0xc000e88e20>: failed to create HCP cluster ea-listhfgh7b: failed waiting for cluster="ea-listhfgh7b" in resourcegroup="ea-list-6p4sr9k5q7bf" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/62436e44-f35d-4583-a471-e3b8c1476144 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/62436e44-f35d-4583-a471-e3b8c1476144", "name": "62436e44-f35d-4583-a471-e3b8c1476144", "status": "Failed", "startTime": "2026-08-25T14:23:30.392692104Z", "endTime": "2026-08-25T14:42:33.727154801Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 13:20:40 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test Unexpected error: <fmt.wrapError \| 0xc0008f8400>: failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-5f7fqf8wr7rh" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/996552c5-cfd8-458f-bb7e-b012f72bb6fd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/996552c5-cfd8-458f-bb7e-b012f72bb6fd", "name": "996552c5-cfd8-458f-bb7e-b012f72bb6fd", "status": "Failed", "startTime": "2026-08-25T14:23:19.305316303Z", "endTime": "2026-08-25T14:42:23.727933134Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test Unexpected error: <fmt.wrapError \| 0xc0008f8400>: failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-5f7fqf8wr7rh" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/996552c5-cfd8-458f-bb7e-b012f72bb6fd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/996552c5-cfd8-458f-bb7e-b012f72bb6fd", "name": "996552c5-cfd8-458f-bb7e-b012f72bb6fd", "status": "Failed", "startTime": "2026-08-25T14:23:19.305316303Z", "endTime": "2026-08-25T14:42:23.727933134Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 13:20:40 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-jfwr5v with version 4.21.30 Unexpected error: <fmt.wrapError \| 0xc001072ac0>: failed to create HCP cluster np-version-upgrade-cluster-jfwr5v: failed waiting for cluster="np-version-upgrade-cluster-jfwr5v" in resourcegroup="rg-np-version-upgrade-jfwr5v-5pf2tksxbxvk" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3f9c0f4c-5e66-444f-99cf-8a9d11c913f9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3f9c0f4c-5e66-444f-99cf-8a9d11c913f9", "name": "3f9c0f4c-5e66-444f-99cf-8a9d11c913f9", "status": "Failed", "startTime": "2026-08-25T14:23:31.574965707Z", "endTime": "2026-08-25T14:42:33.731961599Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-jfwr5v with version 4.21.30 Unexpected error: <fmt.wrapError \| 0xc001072ac0>: failed to create HCP cluster np-version-upgrade-cluster-jfwr5v: failed waiting for cluster="np-version-upgrade-cluster-jfwr5v" in resourcegroup="rg-np-version-upgrade-jfwr5v-5pf2tksxbxvk" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3f9c0f4c-5e66-444f-99cf-8a9d11c913f9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3f9c0f4c-5e66-444f-99cf-8a9d11c913f9", "name": "3f9c0f4c-5e66-444f-99cf-8a9d11c913f9", "status": "Failed", "startTime": "2026-08-25T14:23:31.574965707Z", "endTime": "2026-08-25T14:42:33.731961599Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator, ingress-operator, router]; provider Microsoft.RedHatOpenShift 13:20:40 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-l7z8bjhz25gx/cluster-candidate-4-21-mj7vzs should provision Unexpected error: <fmt.wrapError \| 0xc000690a00>: failed to create HCP cluster cluster-candidate-4-21-mj7vzs: failed waiting for cluster="cluster-candidate-4-21-mj7vzs" in resourcegroup="rg-candidate-4-21-l7z8bjhz25gx" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6b46df85-5514-4dc0-92e8-a7d1f99eb864 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6b46df85-5514-4dc0-92e8-a7d1f99eb864", "name": "6b46df85-5514-4dc0-92e8-a7d1f99eb864", "status": "Failed", "startTime": "2026-08-25T14:23:19.273281186Z", "endTime": "2026-08-25T14:42:23.718787141Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-l7z8bjhz25gx/cluster-candidate-4-21-mj7vzs should provision Unexpected error: <fmt.wrapError \| 0xc000690a00>: failed to create HCP cluster cluster-candidate-4-21-mj7vzs: failed waiting for cluster="cluster-candidate-4-21-mj7vzs" in resourcegroup="rg-candidate-4-21-l7z8bjhz25gx" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6b46df85-5514-4dc0-92e8-a7d1f99eb864 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6b46df85-5514-4dc0-92e8-a7d1f99eb864", "name": "6b46df85-5514-4dc0-92e8-a7d1f99eb864", "status": "Failed", "startTime": "2026-08-25T14:23:19.273281186Z", "endTime": "2026-08-25T14:42:23.718787141Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 13:20:40 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0xc0010c43c0>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-zcf2d6kk52j7" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/208d78c8-6efe-4c95-b106-65d2487b4d6f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/208d78c8-6efe-4c95-b106-65d2487b4d6f", "name": "208d78c8-6efe-4c95-b106-65d2487b4d6f", "status": "Failed", "startTime": "2026-08-25T14:23:10.560757862Z", "endTime": "2026-08-25T14:42:13.708090431Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0xc0010c43c0>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-zcf2d6kk52j7" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/208d78c8-6efe-4c95-b106-65d2487b4d6f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/208d78c8-6efe-4c95-b106-65d2487b4d6f", "name": "208d78c8-6efe-4c95-b106-65d2487b4d6f", "status": "Failed", "startTime": "2026-08-25T14:23:10.560757862Z", "endTime": "2026-08-25T14:42:13.708090431Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager, catalog-operator, hosted-cluster-config-operator, machine-approver, olm-operator, openshift-apiserver, packageserver, router]; provider Microsoft.RedHatOpenShift 13:20:40 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc0011ca240>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-d4mb4xht7542" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d1b31ca4-5e1a-4bef-8eae-3fae81f040ec -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d1b31ca4-5e1a-4bef-8eae-3fae81f040ec", "name": "d1b31ca4-5e1a-4bef-8eae-3fae81f040ec", "status": "Failed", "startTime": "2026-08-25T14:23:29.491781901Z", "endTime": "2026-08-25T14:42:33.748439526Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, cluster-node-tuning-operator, cluster-version-operator, cluster-policy-controller, olm-operator, redhat-operators-catalog, certified-operators-catalog, cluster-image-registry-operator, redhat-marketplace-catalog, cluster-storage-operator, ingress-operator, csi-snapshot-controller-operator, catalog-operator, machine-approver, openshift-route-controller-manager, hosted-cluster-config-operator, azure-cloud-controller-manager, packageserver, openshift-controller-manager, dns-operator, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc0011ca240>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-d4mb4xht7542" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d1b31ca4-5e1a-4bef-8eae-3fae81f040ec -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d1b31ca4-5e1a-4bef-8eae-3fae81f040ec", "name": "d1b31ca4-5e1a-4bef-8eae-3fae81f040ec", "status": "Failed", "startTime": "2026-08-25T14:23:29.491781901Z", "endTime": "2026-08-25T14:42:33.748439526Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, cluster-node-tuning-operator, cluster-version-operator, cluster-policy-controller, olm-operator, redhat-operators-catalog, certified-operators-catalog, cluster-image-registry-operator, redhat-marketplace-catalog, cluster-storage-operator, ingress-operator, csi-snapshot-controller-operator, catalog-operator, machine-approver, openshift-route-controller-manager, hosted-cluster-config-operator, azure-cloud-controller-manager, packageserver, openshift-controller-manager, dns-operator, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift 13:20:40 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:721]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000f4a7e0>: failed to create HCP cluster np-dg-minor-9qfj96: failed waiting for cluster="np-dg-minor-9qfj96" in resourcegroup="rg-np-dg-minor-9qfj96-vfn2xzp2hh4x" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8ce4f4e-f284-4530-a975-e577f84037f8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8ce4f4e-f284-4530-a975-e577f84037f8", "name": "c8ce4f4e-f284-4530-a975-e577f84037f8", "status": "Failed", "startTime": "2026-08-25T14:23:10.502180324Z", "endTime": "2026-08-25T14:42:13.711516452Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:721]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000f4a7e0>: failed to create HCP cluster np-dg-minor-9qfj96: failed waiting for cluster="np-dg-minor-9qfj96" in resourcegroup="rg-np-dg-minor-9qfj96-vfn2xzp2hh4x" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8ce4f4e-f284-4530-a975-e577f84037f8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8ce4f4e-f284-4530-a975-e577f84037f8", "name": "c8ce4f4e-f284-4530-a975-e577f84037f8", "status": "Failed", "startTime": "2026-08-25T14:23:10.502180324Z", "endTime": "2026-08-25T14:42:13.711516452Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 13:20:40 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1 Unexpected error: <fmt.wrapError \| 0xc0010e6ae0>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-vrnlz7rb6wl4" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8f07a9ee-2395-4d42-aa33-38519e85c49d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8f07a9ee-2395-4d42-aa33-38519e85c49d", "name": "8f07a9ee-2395-4d42-aa33-38519e85c49d", "status": "Failed", "startTime": "2026-08-25T14:37:18.839720203Z", "endTime": "2026-08-25T14:56:23.617637004Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1 Unexpected error: <fmt.wrapError \| 0xc0010e6ae0>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-vrnlz7rb6wl4" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8f07a9ee-2395-4d42-aa33-38519e85c49d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8f07a9ee-2395-4d42-aa33-38519e85c49d", "name": "8f07a9ee-2395-4d42-aa33-38519e85c49d", "status": "Failed", "startTime": "2026-08-25T14:37:18.839720203Z", "endTime": "2026-08-25T14:56:23.617637004Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, router]; provider Microsoft.RedHatOpenShift 13:20:40 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000fa6640>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-97gvpsltv9sn" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/55ca21a5-264c-4b89-812c-1bd654f8f905 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/55ca21a5-264c-4b89-812c-1bd654f8f905", "name": "55ca21a5-264c-4b89-812c-1bd654f8f905", "status": "Failed", "startTime": "2026-08-25T14:24:51.992542762Z", "endTime": "2026-08-25T14:43:53.990674833Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000fa6640>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-97gvpsltv9sn" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/55ca21a5-264c-4b89-812c-1bd654f8f905 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/55ca21a5-264c-4b89-812c-1bd654f8f905", "name": "55ca21a5-264c-4b89-812c-1bd654f8f905", "status": "Failed", "startTime": "2026-08-25T14:24:51.992542762Z", "endTime": "2026-08-25T14:43:53.990674833Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator]; provider Microsoft.RedHatOpenShift 13:20:40 UTC · failed at e2e · test SRE should return 409 when boot diagnostics is disabled on a VM · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:417]: failed to create HCP cluster "sre-hcp-cluster-bootdiag" for boot diagnostics test Unexpected error: <fmt.wrapError \| 0xc000a922c0>: failed to create HCP cluster sre-hcp-cluster-bootdiag: failed waiting for cluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-4m6h5kd6h7vt" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/acbfe6bf-3ab4-49b1-a7fb-9d29f2abc4ed -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/acbfe6bf-3ab4-49b1-a7fb-9d29f2abc4ed", "name": "acbfe6bf-3ab4-49b1-a7fb-9d29f2abc4ed", "status": "Failed", "startTime": "2026-08-25T14:23:09.115641816Z", "endTime": "2026-08-25T14:42:13.649502526Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:417]: failed to create HCP cluster "sre-hcp-cluster-bootdiag" for boot diagnostics test Unexpected error: <fmt.wrapError \| 0xc000a922c0>: failed to create HCP cluster sre-hcp-cluster-bootdiag: failed waiting for cluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-4m6h5kd6h7vt" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/acbfe6bf-3ab4-49b1-a7fb-9d29f2abc4ed -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/acbfe6bf-3ab4-49b1-a7fb-9d29f2abc4ed", "name": "acbfe6bf-3ab4-49b1-a7fb-9d29f2abc4ed", "status": "Failed", "startTime": "2026-08-25T14:23:09.115641816Z", "endTime": "2026-08-25T14:42:13.649502526Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 13:20:40 UTC · failed at e2e · test FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc0009158e0>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-stcdz6kncxhr" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/567d6c13-28cc-4446-ab15-21bd709a116c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/567d6c13-28cc-4446-ab15-21bd709a116c", "name": "567d6c13-28cc-4446-ab15-21bd709a116c", "status": "Failed", "startTime": "2026-08-25T14:23:08.741636847Z", "endTime": "2026-08-25T14:42:13.719295306Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc0009158e0>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-stcdz6kncxhr" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/567d6c13-28cc-4446-ab15-21bd709a116c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/567d6c13-28cc-4446-ab15-21bd709a116c", "name": "567d6c13-28cc-4446-ab15-21bd709a116c", "status": "Failed", "startTime": "2026-08-25T14:23:08.741636847Z", "endTime": "2026-08-25T14:42:13.719295306Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 13:20:40 UTC · failed at e2e · test SRE can pause schedules to stop backup execution for an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc0011468e0>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-w7786nk5nznp" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9975191b-6ab2-42a8-85c6-83aed94ceaf0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9975191b-6ab2-42a8-85c6-83aed94ceaf0", "name": "9975191b-6ab2-42a8-85c6-83aed94ceaf0", "status": "Failed", "startTime": "2026-08-25T14:23:19.963379665Z", "endTime": "2026-08-25T14:42:23.660042904Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc0011468e0>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-w7786nk5nznp" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9975191b-6ab2-42a8-85c6-83aed94ceaf0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9975191b-6ab2-42a8-85c6-83aed94ceaf0", "name": "9975191b-6ab2-42a8-85c6-83aed94ceaf0", "status": "Failed", "startTime": "2026-08-25T14:23:19.963379665Z", "endTime": "2026-08-25T14:42:23.660042904Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 13:20:40 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-fsbfnssnhqkf/cluster-candidate-5-0-2knmvc should provision Unexpected error: <fmt.wrapError \| 0xc000e46260>: failed to create HCP cluster cluster-candidate-5-0-2knmvc: failed waiting for cluster="cluster-candidate-5-0-2knmvc" in resourcegroup="rg-candidate-5-0-fsbfnssnhqkf" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cc01a940-8988-427e-9d74-c468fbaf0fee -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cc01a940-8988-427e-9d74-c468fbaf0fee", "name": "cc01a940-8988-427e-9d74-c468fbaf0fee", "status": "Failed", "startTime": "2026-08-25T14:23:19.159652548Z", "endTime": "2026-08-25T14:42:23.741564783Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-fsbfnssnhqkf/cluster-candidate-5-0-2knmvc should provision Unexpected error: <fmt.wrapError \| 0xc000e46260>: failed to create HCP cluster cluster-candidate-5-0-2knmvc: failed waiting for cluster="cluster-candidate-5-0-2knmvc" in resourcegroup="rg-candidate-5-0-fsbfnssnhqkf" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cc01a940-8988-427e-9d74-c468fbaf0fee -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cc01a940-8988-427e-9d74-c468fbaf0fee", "name": "cc01a940-8988-427e-9d74-c468fbaf0fee", "status": "Failed", "startTime": "2026-08-25T14:23:19.159652548Z", "endTime": "2026-08-25T14:42:23.741564783Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 13:20:40 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc000b14a20>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-ld6hfp9nvt27" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/df760612-adee-4fe6-b2cc-01751b699824 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/df760612-adee-4fe6-b2cc-01751b699824", "name": "df760612-adee-4fe6-b2cc-01751b699824", "status": "Failed", "startTime": "2026-08-25T14:23:30.466260682Z", "endTime": "2026-08-25T14:42:33.758671656Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator, olm-operator, openshift-route-controller-manager, cluster-storage-operator, catalog-operator, certified-operators-catalog, hosted-cluster-config-operator, csi-snapshot-controller-operator, community-operators-catalog, cluster-image-registry-operator, cluster-version-operator, openshift-controller-manager, cluster-policy-controller, redhat-operators-catalog, packageserver, cluster-network-operator, dns-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 2 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 2 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc000b14a20>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-ld6hfp9nvt27" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/df760612-adee-4fe6-b2cc-01751b699824 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/df760612-adee-4fe6-b2cc-01751b699824", "name": "df760612-adee-4fe6-b2cc-01751b699824", "status": "Failed", "startTime": "2026-08-25T14:23:30.466260682Z", "endTime": "2026-08-25T14:42:33.758671656Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator, olm-operator, openshift-route-controller-manager, cluster-storage-operator, catalog-operator, certified-operators-catalog, hosted-cluster-config-operator, csi-snapshot-controller-operator, community-operators-catalog, cluster-image-registry-operator, cluster-version-operator, openshift-controller-manager, cluster-policy-controller, redhat-operators-catalog, packageserver, cluster-network-operator, dns-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 2 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 2 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator, cluster-storage-operator, csi-snapshot-controller-operator, dns-operator, ingress-operator, packageserver]; provider Microsoft.RedHatOpenShift 13:20:40 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:147]: failed to create HCP cluster "cp-ystream-upgrade-4-21-qclt2j" at install version 4.20 Unexpected error: <fmt.wrapError \| 0xc0012c2120>: failed to create HCP cluster cp-ystream-upgrade-4-21-qclt2j: failed waiting for cluster="cp-ystream-upgrade-4-21-qclt2j" in resourcegroup="rg-cp-ystream-upgrade-4-21-2mbkqtdbwbbj" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c79cbefa-a0f4-4710-a616-f42a2f2efab3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c79cbefa-a0f4-4710-a616-f42a2f2efab3", "name": "c79cbefa-a0f4-4710-a616-f42a2f2efab3", "status": "Failed", "startTime": "2026-08-25T14:23:09.836508695Z", "endTime": "2026-08-25T14:42:13.731155696Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: packageserver, dns-operator, csi-snapshot-controller-operator, cluster-storage-operator, ingress-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, packageserver deployment has 2 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:147]: failed to create HCP cluster "cp-ystream-upgrade-4-21-qclt2j" at install version 4.20 Unexpected error: <fmt.wrapError \| 0xc0012c2120>: failed to create HCP cluster cp-ystream-upgrade-4-21-qclt2j: failed waiting for cluster="cp-ystream-upgrade-4-21-qclt2j" in resourcegroup="rg-cp-ystream-upgrade-4-21-2mbkqtdbwbbj" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c79cbefa-a0f4-4710-a616-f42a2f2efab3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c79cbefa-a0f4-4710-a616-f42a2f2efab3", "name": "c79cbefa-a0f4-4710-a616-f42a2f2efab3", "status": "Failed", "startTime": "2026-08-25T14:23:09.836508695Z", "endTime": "2026-08-25T14:42:13.731155696Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: packageserver, dns-operator, csi-snapshot-controller-operator, cluster-storage-operator, ingress-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, packageserver deployment has 2 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift 13:20:40 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc000bbcde0>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-ckzdqmgrqb4b" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9c26091f-a379-4334-aa6c-e0a5fcf1a616 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9c26091f-a379-4334-aa6c-e0a5fcf1a616", "name": "9c26091f-a379-4334-aa6c-e0a5fcf1a616", "status": "Failed", "startTime": "2026-08-25T14:25:28.457957025Z", "endTime": "2026-08-25T14:44:33.709034069Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc000bbcde0>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-ckzdqmgrqb4b" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9c26091f-a379-4334-aa6c-e0a5fcf1a616 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9c26091f-a379-4334-aa6c-e0a5fcf1a616", "name": "9c26091f-a379-4334-aa6c-e0a5fcf1a616", "status": "Failed", "startTime": "2026-08-25T14:25:28.457957025Z", "endTime": "2026-08-25T14:44:33.709034069Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, packageserver, router]; provider Microsoft.RedHatOpenShift 13:20:40 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-cgzvsfdzkmd6/cluster-candidate-4-20-gm7v89 should provision Unexpected error: <fmt.wrapError \| 0xc00146c980>: failed to create HCP cluster cluster-candidate-4-20-gm7v89: failed waiting for cluster="cluster-candidate-4-20-gm7v89" in resourcegroup="rg-candidate-4-20-cgzvsfdzkmd6" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/048ffc99-876d-41b4-8fb3-df2166c912e4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/048ffc99-876d-41b4-8fb3-df2166c912e4", "name": "048ffc99-876d-41b4-8fb3-df2166c912e4", "status": "Failed", "startTime": "2026-08-25T14:23:19.870414032Z", "endTime": "2026-08-25T14:42:23.711491128Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, ingress-operator, cluster-image-registry-operator, dns-operator, hosted-cluster-config-operator, packageserver, catalog-operator, cluster-version-operator, cluster-network-operator, csi-snapshot-controller-operator, cluster-policy-controller, cluster-node-tuning-operator, olm-operator, redhat-marketplace-catalog, openshift-route-controller-manager, cluster-storage-operator, redhat-operators-catalog, community-operators-catalog, certified-operators-catalog, machine-approver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-cgzvsfdzkmd6/cluster-candidate-4-20-gm7v89 should provision Unexpected error: <fmt.wrapError \| 0xc00146c980>: failed to create HCP cluster cluster-candidate-4-20-gm7v89: failed waiting for cluster="cluster-candidate-4-20-gm7v89" in resourcegroup="rg-candidate-4-20-cgzvsfdzkmd6" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/048ffc99-876d-41b4-8fb3-df2166c912e4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/048ffc99-876d-41b4-8fb3-df2166c912e4", "name": "048ffc99-876d-41b4-8fb3-df2166c912e4", "status": "Failed", "startTime": "2026-08-25T14:23:19.870414032Z", "endTime": "2026-08-25T14:42:23.711491128Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, ingress-operator, cluster-image-registry-operator, dns-operator, hosted-cluster-config-operator, packageserver, catalog-operator, cluster-version-operator, cluster-network-operator, csi-snapshot-controller-operator, cluster-policy-controller, cluster-node-tuning-operator, olm-operator, redhat-marketplace-catalog, openshift-route-controller-manager, cluster-storage-operator, redhat-operators-catalog, community-operators-catalog, certified-operators-catalog, machine-approver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 13:20:40 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--kcshtglk78v4/cluster-hs-pre-j976nz should provision Unexpected error: <fmt.wrapError \| 0xc001828400>: failed to create HCP cluster cluster-hs-pre-j976nz: failed waiting for cluster="cluster-hs-pre-j976nz" in resourcegroup="rg-hs-pre--kcshtglk78v4" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/43093e9e-e93e-48ed-9142-17883753654a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/43093e9e-e93e-48ed-9142-17883753654a", "name": "43093e9e-e93e-48ed-9142-17883753654a", "status": "Failed", "startTime": "2026-08-25T14:23:19.671981847Z", "endTime": "2026-08-25T14:42:23.717184086Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--kcshtglk78v4/cluster-hs-pre-j976nz should provision Unexpected error: <fmt.wrapError \| 0xc001828400>: failed to create HCP cluster cluster-hs-pre-j976nz: failed waiting for cluster="cluster-hs-pre-j976nz" in resourcegroup="rg-hs-pre--kcshtglk78v4" to finish creating: GET https://rp.j9549056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/43093e9e-e93e-48ed-9142-17883753654a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/43093e9e-e93e-48ed-9142-17883753654a", "name": "43093e9e-e93e-48ed-9142-17883753654a", "status": "Failed", "startTime": "2026-08-25T14:23:19.671981847Z", "endTime": "2026-08-25T14:42:23.717184086Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (2) alert [hcp] KubePodNotReady fired 13:20:40 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 58 time(s) (19 unknown, 39 known) Firing 1: State: Resolved Started: 2026-08-25T14:37:24Z Ended: 2026-08-25T14:50:29Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9549056-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlegoo9bd727bjcnuetqvupvv32qu8-p0k5e8q6m7g0x4g", pod="router-6d67c7888f-hqj2v", severity="warning" Description: Pod ocm-arohcpci01-2sdlegoo9bd727bjcnuetqvupvv32qu8-p0k5e8q6m7g0x4g/router-6d67c7888f-hqj2v has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-08-25T14:37:24Z Ended: 2026-08-25T14:43:24Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9549056-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlejejkoqkg3lci2l992u8dnigrvj2-p9y5g4g2q1m8z4y", pod="router-cf94d77fb-gjndj", severity="warning" Description: Pod ocm-arohcpci01-2sdlejejkoqkg3lci2l992u8dnigrvj2-p9y5g4g2q1m8z4y/router-cf94d77fb-gjndj has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-08-25T14:37:24Z Ended: 2026-08-25T14:50:29Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9549056-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlegoo9bd727bjcnuetqvupvv32qu8-p0k5e8q6m7g0x4g", pod="router-6d67c7888f-9jf9z", severity="warning" Description: Pod ocm-arohcpci01-2sdlegoo9bd727bjcnuetqvupvv32qu8-p0k5e8q6m7g0x4g/router-6d67c7888f-9jf9z has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-08-25T14:39:24Z Ended: 2026-08-25T14:45:28Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9549056-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlejsuuod9gocuvio5n9l642vqd4eh-x6h6n4p7y9s8w9d", pod="dns-operator-6c6b4fb7cd-pfdnm", severity="warning" Description: Pod ocm-arohcpci01-2sdlejsuuod9gocuvio5n9l642vqd4eh-x6h6n4p7y9s8w9d/dns-operator-6c6b4fb7cd-pfdnm has been in a non-ready state for longer than 5 minutes. Firing 5: State: Resolved Started: 2026-08-25T14:39:24Z Ended: 2026-08-25T14:51:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9549056-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlem78mrdia1hmshmk1is9cukdlrjd-t6l2v9o7c8z1u3c", pod="router-5667b96f5d-42xln", severity="warning" Description: Pod ocm-arohcpci01-2sdlem78mrdia1hmshmk1is9cukdlrjd-t6l2v9o7c8z1u3c/router-5667b96f5d-42xln has been in a non-ready state for longer than 5 minutes. Firing 6: State: Resolved Started: 2026-08-25T14:39:24Z Ended: 2026-08-25T14:45:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9549056-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlejsuuod9gocuvio5n9l642vqd4eh-x6h6n4p7y9s8w9d", pod="cluster-image-registry-operator-5656f84f95-g7lnk", severity="warning" Description: Pod ocm-arohcpci01-2sdlejsuuod9gocuvio5n9l642vqd4eh-x6h6n4p7y9s8w9d/cluster-image-registry-operator-5656f84f95-g7lnk has been in a non-ready state for longer than 5 minutes. Firing 7: State: Resolved Started: 2026-08-25T14:39:24Z Ended: 2026-08-25T14:45:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9549056-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlfd1jcj2cm3mnt32pp6a788r14tqr-a3w9p7o0b2m6y7d", pod="router-687b49f995-ztm4t", severity="warning" Description: Pod ocm-arohcpci01-2sdlfd1jcj2cm3mnt32pp6a788r14tqr-a3w9p7o0b2m6y7d/router-687b49f995-ztm4t has been in a non-ready state for longer than 5 minutes. Firing 8: State: Resolved Started: 2026-08-25T14:39:24Z Ended: 2026-08-25T14:51:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9549056-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlem6anaqvl6nuptvbhspigoa8pi7f-q7i5x4f3a4u8u4x", pod="router-c5b76f6b7-vbmjg", severity="warning" Description: Pod ocm-arohcpci01-2sdlem6anaqvl6nuptvbhspigoa8pi7f-q7i5x4f3a4u8u4x/router-c5b76f6b7-vbmjg has been in a non-ready state for longer than 5 minutes. Firing 9: State: Resolved Started: 2026-08-25T14:39:24Z Ended: 2026-08-25T14:46:25Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9549056-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlejsuuod9gocuvio5n9l642vqd4eh-x6h6n4p7y9s8w9d", pod="ingress-operator-687f786bc9-67mkc", severity="warning" Description: Pod ocm-arohcpci01-2sdlejsuuod9gocuvio5n9l642vqd4eh-x6h6n4p7y9s8w9d/ingress-operator-687f786bc9-67mkc has been in a non-ready state for longer than 5 minutes. Firing 10: State: Resolved Started: 2026-08-25T14:39:24Z Ended: 2026-08-25T14:45:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9549056-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdleopo6msip9htu3nm9s74075vsivg-f2a3p1v9l8c5q7c", pod="dns-operator-5655b66c99-ltgth", severity="warning" Description: Pod ocm-arohcpci01-2sdleopo6msip9htu3nm9s74075vsivg-f2a3p1v9l8c5q7c/dns-operator-5655b66c99-ltgth has been in a non-ready state for longer than 5 minutes. Firing 11: State: Resolved Started: 2026-08-25T14:39:24Z Ended: 2026-08-25T14:45:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9549056-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdleopo6msip9htu3nm9s74075vsivg-f2a3p1v9l8c5q7c", pod="cluster-image-registry-operator-6b6c7779fc-7ws99", severity="warning" Description: Pod ocm-arohcpci01-2sdleopo6msip9htu3nm9s74075vsivg-f2a3p1v9l8c5q7c/cluster-image-registry-operator-6b6c7779fc-7ws99 has been in a non-ready state for longer than 5 minutes. Firing 12: State: Resolved Started: 2026-08-25T14:39:24Z Ended: 2026-08-25T14:45:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9549056-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdleopo6msip9htu3nm9s74075vsivg-f2a3p1v9l8c5q7c", pod="ingress-operator-6bd6477fcc-lxkk4", severity="warning" Description: Pod ocm-arohcpci01-2sdleopo6msip9htu3nm9s74075vsivg-f2a3p1v9l8c5q7c/ingress-operator-6bd6477fcc-lxkk4 has been in a non-ready state for longer than 5 minutes. Firing 13: State: Resolved Started: 2026-08-25T14:39:24Z Ended: 2026-08-25T14:47:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9549056-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdleolvj8jkpoukleh4otfco23bj7gk-v20260901", pod="router-5f7bcbb5c8-hqd9h", severity="warning" Description: Pod ocm-arohcpci01-2sdleolvj8jkpoukleh4otfco23bj7gk-v20260901/router-5f7bcbb5c8-hqd9h has been in a non-ready state for longer than 5 minutes. Firing 14: State: Resolved Started: 2026-08-25T14:40:25Z Ended: 2026-08-25T14:52:24Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9549056-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlf3e90766j6dqef5ebgosi6grm0nf-private-kas", pod="router-645d7f96c4-m7mfv", severity="warning" Description: Pod ocm-arohcpci01-2sdlf3e90766j6dqef5ebgosi6grm0nf-private-kas/router-645d7f96c4-m7mfv has been in a non-ready state for longer than 5 minutes. Firing 15: State: Resolved Started: 2026-08-25T14:41:24Z Ended: 2026-08-25T14:48:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9549056-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlem6anaqvl6nuptvbhspigoa8pi7f-q7i5x4f3a4u8u4x", pod="ingress-operator-5dbd9fdbb6-lk864", severity="warning" Description: Pod ocm-arohcpci01-2sdlem6anaqvl6nuptvbhspigoa8pi7f-q7i5x4f3a4u8u4x/ingress-operator-5dbd9fdbb6-lk864 has been in a non-ready state for longer than 5 minutes. Firing 16: State: Resolved Started: 2026-08-25T14:41:25Z Ended: 2026-08-25T14:48:28Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9549056-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlem6anaqvl6nuptvbhspigoa8pi7f-q7i5x4f3a4u8u4x", pod="cluster-network-operator-7c7f7f8b99-5kwm5", severity="warning" Description: Pod ocm-arohcpci01-2sdlem6anaqvl6nuptvbhspigoa8pi7f-q7i5x4f3a4u8u4x/cluster-network-operator-7c7f7f8b99-5kwm5 has been in a non-ready state for longer than 5 minutes. Firing 17: State: Resolved Started: 2026-08-25T14:41:25Z Ended: 2026-08-25T14:51:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9549056-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlem4jmsj57unhcuuumlc6abnimmop-v7e8k2i7y9u0b2p", pod="router-58d8d6898f-k9nqv", severity="warning" Description: Pod ocm-arohcpci01-2sdlem4jmsj57unhcuuumlc6abnimmop-v7e8k2i7y9u0b2p/router-58d8d6898f-k9nqv has been in a non-ready state for longer than 5 minutes. Firing 18: State: Resolved Started: 2026-08-25T14:41:25Z Ended: 2026-08-25T14:51:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9549056-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdleop0e81vroks1mrdm6r5jl79961q-q1n9m0x8z5a8w9z", pod="router-5c8d77f9c7-wxfgv", severity="warning" Description: Pod ocm-arohcpci01-2sdleop0e81vroks1mrdm6r5jl79961q-q1n9m0x8z5a8w9z/router-5c8d77f9c7-wxfgv has been in a non-ready state for longer than 5 minutes. Firing 19: State: Resolved Started: 2026-08-25T14:42:24Z Ended: 2026-08-25T14:51:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9549056-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdlem3cfjc7gd3ldptserbelahoc1dp-f9m3o7e8f8j6d7j", pod="router-6d8949db64-nmdv8", severity="warning" Description: Pod ocm-arohcpci01-2sdlem3cfjc7gd3ldptserbelahoc1dp-f9m3o7e8f8j6d7j/router-6d8949db64-nmdv8 has been in a non-ready state for longer than 5 minutes. alert [svc] UJClusterProvisionErrors6h30m fired 13:20:40 UTC · failed at alert · test [aro-hcp-observability] [svc] alert UJClusterProvisionErrors6h30m does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-08-25T15:14:00Z Severity: Sev3 Labels: alertname="UJClusterProvisionErrors6h30m", cluster="ci01-j9549056-svc", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9549056/providers/microsoft.monitor/accounts/services-j9549056", severity="3", short_window="30m", slo="cluster-provision-errors" Description: More than 30% of cluster create (install) operations are in failed state sustained over 30 minutes, indicating a medium error budget burn (6x) that would exhaust the 95% SLO budget in ~28 hours.
13:18:54	pull-ci-Azure-ARO-HCP-main-e2e-parallel head b7448d3	e2e	failed	#6657 (open)	1	e2e (1) cluster entered terminal Failed state after role assignment deployment 13:18:54 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.001s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.001s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ...
12:56:26	pull-ci-Azure-ARO-HCP-main-e2e-parallel head b89541a	e2e	failed	#6670 (open)	1	e2e (1) ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 12:56:26 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling Unexpected error: <fmt.wrapError \| 0xc001048d20>: failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-pmmkk7vxjzf8" to finish creating: GET https://rp.j6018176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ad1b70b6-b35f-4112-9ae9-1df6c2649ab2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ad1b70b6-b35f-4112-9ae9-1df6c2649ab2", "name": "ad1b70b6-b35f-4112-9ae9-1df6c2649ab2", "status": "Failed", "startTime": "2026-08-25T14:00:12.864408043Z", "endTime": "2026-08-25T14:19:12.422414092Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling Unexpected error: <fmt.wrapError \| 0xc001048d20>: failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-pmmkk7vxjzf8" to finish creating: GET https://rp.j6018176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ad1b70b6-b35f-4112-9ae9-1df6c2649ab2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ad1b70b6-b35f-4112-9ae9-1df6c2649ab2", "name": "ad1b70b6-b35f-4112-9ae9-1df6c2649ab2", "status": "Failed", "startTime": "2026-08-25T14:00:12.864408043Z", "endTime": "2026-08-25T14:19:12.422414092Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred
12:43:00	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 9749b2d	e2e	failed	#4566 (open)	3	e2e (3) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 12:43:00 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc0009084e0>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-qbnbmqvjsdcq" to finish creating: GET https://rp.j8712960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0cf3357-f80d-46bd-801d-3a7369bbca29 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0cf3357-f80d-46bd-801d-3a7369bbca29", "name": "f0cf3357-f80d-46bd-801d-3a7369bbca29", "status": "Failed", "startTime": "2026-08-25T13:45:24.403678541Z", "endTime": "2026-08-25T14:04:32.105380072Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc0009084e0>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-qbnbmqvjsdcq" to finish creating: GET https://rp.j8712960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0cf3357-f80d-46bd-801d-3a7369bbca29 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0cf3357-f80d-46bd-801d-3a7369bbca29", "name": "f0cf3357-f80d-46bd-801d-3a7369bbca29", "status": "Failed", "startTime": "2026-08-25T13:45:24.403678541Z", "endTime": "2026-08-25T14:04:32.105380072Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 12:43:00 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc000ee5280>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-kxmgj9sxlkxj" to finish creating: GET https://rp.j8712960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/33987596-6be7-4dfe-aa71-8328c20f9b13 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/33987596-6be7-4dfe-aa71-8328c20f9b13", "name": "33987596-6be7-4dfe-aa71-8328c20f9b13", "status": "Failed", "startTime": "2026-08-25T13:44:21.781571493Z", "endTime": "2026-08-25T14:03:22.119943491Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc000ee5280>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-kxmgj9sxlkxj" to finish creating: GET https://rp.j8712960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/33987596-6be7-4dfe-aa71-8328c20f9b13 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/33987596-6be7-4dfe-aa71-8328c20f9b13", "name": "33987596-6be7-4dfe-aa71-8328c20f9b13", "status": "Failed", "startTime": "2026-08-25T13:44:21.781571493Z", "endTime": "2026-08-25T14:03:22.119943491Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 12:43:00 UTC · failed at e2e · test SRE can pause schedules to stop backup execution for an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc0018f4220>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-t52sl5qhgrmd" to finish creating: GET https://rp.j8712960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b14f096f-1fdb-4e55-b7c3-07232f031770 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b14f096f-1fdb-4e55-b7c3-07232f031770", "name": "b14f096f-1fdb-4e55-b7c3-07232f031770", "status": "Failed", "startTime": "2026-08-25T13:43:02.409731764Z", "endTime": "2026-08-25T14:02:02.069640925Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc0018f4220>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-t52sl5qhgrmd" to finish creating: GET https://rp.j8712960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b14f096f-1fdb-4e55-b7c3-07232f031770 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b14f096f-1fdb-4e55-b7c3-07232f031770", "name": "b14f096f-1fdb-4e55-b7c3-07232f031770", "status": "Failed", "startTime": "2026-08-25T13:43:02.409731764Z", "endTime": "2026-08-25T14:02:02.069640925Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred
12:42:48	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 669f42e	n/a	passed	#6618 (open)	0	n/a
12:17:19	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 5c0ce63	n/a	passed	#6669 (open)	0	n/a
12:12:42	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 3c81c74	n/a	passed	#6645 (open)	0	n/a
11:59:52	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 0b3e24f	n/a	passed	#6676 (open)	0	n/a
11:52:53	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 1460cb0	unknown	failed	#6676 (open)	1	non-artifact-backed failure (no junit artifacts)
11:40:45	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 515e34b	e2e	failed	#6583 (open)	1	e2e (1) ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift 11:40:45 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc001ac6140>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-xsr5g7ds2s49" to finish creating: GET https://rp.j7467264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3c63b46d-df4e-4caf-b1a9-fee835ae340b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3c63b46d-df4e-4caf-b1a9-fee835ae340b", "name": "3c63b46d-df4e-4caf-b1a9-fee835ae340b", "status": "Failed", "startTime": "2026-08-25T12:28:28.768318189Z", "endTime": "2026-08-25T12:47:36.715383913Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc001ac6140>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-xsr5g7ds2s49" to finish creating: GET https://rp.j7467264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3c63b46d-df4e-4caf-b1a9-fee835ae340b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3c63b46d-df4e-4caf-b1a9-fee835ae340b", "name": "3c63b46d-df4e-4caf-b1a9-fee835ae340b", "status": "Failed", "startTime": "2026-08-25T12:28:28.768318189Z", "endTime": "2026-08-25T12:47:36.715383913Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred
11:14:00	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 5fdef9d	e2e	failed	★#6364 (open)	1	e2e (1) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift 11:14:00 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:93]: failed to create HCP cluster "sre-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000a6cce0>: failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-pcmvmmpvpflg" to finish creating: GET https://rp.j9594112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/52000f23-e3aa-465f-b2e6-2408f772e24f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/52000f23-e3aa-465f-b2e6-2408f772e24f", "name": "52000f23-e3aa-465f-b2e6-2408f772e24f", "status": "Failed", "startTime": "2026-08-25T11:59:13.55636864Z", "endTime": "2026-08-25T12:18:21.188692138Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:93]: failed to create HCP cluster "sre-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000a6cce0>: failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-pcmvmmpvpflg" to finish creating: GET https://rp.j9594112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/52000f23-e3aa-465f-b2e6-2408f772e24f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/52000f23-e3aa-465f-b2e6-2408f772e24f", "name": "52000f23-e3aa-465f-b2e6-2408f772e24f", "status": "Failed", "startTime": "2026-08-25T11:59:13.55636864Z", "endTime": "2026-08-25T12:18:21.188692138Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred
11:13:43	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 0e91a6e · merge 0e91a6e	n/a	passed	#6675 (merged)	0	n/a
11:03:27	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 69272e6	n/a	passed	#6674 (open)	0	n/a
10:30:42	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 9ec6505	alert	failed	#6433 (open)	3	alert (3) alert [svc] KubeDaemonSetRolloutStuck fired 10:30:42 UTC · failed at alert · test [aro-hcp-observability] [svc] alert KubeDaemonSetRolloutStuck does not fire · suite aro-hcp-tests Show raw failure alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-08-25T11:37:26Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j1136640-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-csi-driver", endpoint="http", environment="ci01", instance="10.128.64.222:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1136640/providers/microsoft.monitor/accounts/services-j1136640", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wwh5l", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet kube-system/aks-secrets-store-csi-driver has not finished or progressed for at least 15 minutes. Firing 2: State: Fired (not resolved) Started: 2026-08-25T11:37:26Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j1136640-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-csi-driver", endpoint="http", environment="ci01", instance="10.128.64.222:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1136640/providers/microsoft.monitor/accounts/services-j1136640", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wwh5l", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet kube-system/aks-secrets-store-csi-driver has not finished or progressed for at least 15 minutes. Firing 3: State: Fired (not resolved) Started: 2026-08-25T11:37:26Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j1136640-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="ama-metrics-node", endpoint="http", environment="ci01", instance="10.128.64.222:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1136640/providers/microsoft.monitor/accounts/services-j1136640", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wwh5l", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet kube-system/ama-metrics-node has not finished or progressed for at least 15 minutes. Firing 4: State: Fired (not resolved) Started: 2026-08-25T11:37:26Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j1136640-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="ama-metrics-node", endpoint="http", environment="ci01", instance="10.128.64.222:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1136640/providers/microsoft.monitor/accounts/services-j1136640", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wwh5l", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet kube-system/ama-metrics-node has not finished or progressed for at least 15 minutes. alert [svc] KubeNodeNotReady fired 10:30:42 UTC · failed at alert · test [aro-hcp-observability] [svc] alert KubeNodeNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) Firing 1: State: Fired (not resolved) Started: 2026-08-25T11:52:32Z Severity: Sev3 Labels: alertname="KubeNodeNotReady", cluster="ci01-j1136640-mgmt-1", component="kubernetes-infrastructure", condition="Ready", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.222:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1136640/providers/microsoft.monitor/accounts/services-j1136640", namespace="prometheus", node="aks-system-30244119-vmss000000", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wwh5l", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", status="true" Description: aks-system-30244119-vmss000000 has been unready for more than 30 minutes. Firing 2: State: Fired (not resolved) Started: 2026-08-25T11:52:32Z Severity: Sev3 Labels: alertname="KubeNodeNotReady", cluster="ci01-j1136640-mgmt-1", component="kubernetes-infrastructure", condition="Ready", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.222:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1136640/providers/microsoft.monitor/accounts/services-j1136640", namespace="prometheus", node="aks-system-30244119-vmss000000", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wwh5l", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", status="true" Description: aks-system-30244119-vmss000000 has been unready for more than 30 minutes. alert [svc] KubeContainerWaiting fired 10:30:42 UTC · failed at alert · test [aro-hcp-observability] [svc] alert KubeContainerWaiting does not fire · suite aro-hcp-tests Show raw failure alert fired 8 time(s) Firing 1: State: Fired (not resolved) Started: 2026-08-25T12:22:23Z Severity: Sev3 Labels: alertname="KubeContainerWaiting", cluster="ci01-j1136640-mgmt-1", component="kubernetes-infrastructure", container="liveness-probe", namespace="kube-system", pod="aks-secrets-store-csi-driver-2k5cw", severity="warning" Description: pod/aks-secrets-store-csi-driver-2k5cw in namespace kube-system on container liveness-probe has been in waiting state for longer than 1 hour. Firing 2: State: Fired (not resolved) Started: 2026-08-25T12:22:23Z Severity: Sev3 Labels: alertname="KubeContainerWaiting", cluster="ci01-j1136640-mgmt-1", component="kubernetes-infrastructure", container="trivy-scanner", namespace="kube-system", pod="eraser-aks-system-30244119-vmss000000-7fwv2", severity="warning" Description: pod/eraser-aks-system-30244119-vmss000000-7fwv2 in namespace kube-system on container trivy-scanner has been in waiting state for longer than 1 hour. Firing 3: State: Fired (not resolved) Started: 2026-08-25T12:22:23Z Severity: Sev3 Labels: alertname="KubeContainerWaiting", cluster="ci01-j1136640-mgmt-1", component="kubernetes-infrastructure", container="prometheus-collector", namespace="kube-system", pod="ama-metrics-node-rvqrk", severity="warning" Description: pod/ama-metrics-node-rvqrk in namespace kube-system on container prometheus-collector has been in waiting state for longer than 1 hour. Firing 4: State: Fired (not resolved) Started: 2026-08-25T12:22:23Z Severity: Sev3 Labels: alertname="KubeContainerWaiting", cluster="ci01-j1136640-mgmt-1", component="kubernetes-infrastructure", container="secrets-store", namespace="kube-system", pod="aks-secrets-store-csi-driver-2k5cw", severity="warning" Description: pod/aks-secrets-store-csi-driver-2k5cw in namespace kube-system on container secrets-store has been in waiting state for longer than 1 hour. Firing 5: State: Fired (not resolved) Started: 2026-08-25T12:22:23Z Severity: Sev3 Labels: alertname="KubeContainerWaiting", cluster="ci01-j1136640-mgmt-1", component="kubernetes-infrastructure", container="addon-token-adapter", namespace="kube-system", pod="ama-metrics-node-rvqrk", severity="warning" Description: pod/ama-metrics-node-rvqrk in namespace kube-system on container addon-token-adapter has been in waiting state for longer than 1 hour. Firing 6: State: Fired (not resolved) Started: 2026-08-25T12:22:23Z Severity: Sev3 Labels: alertname="KubeContainerWaiting", cluster="ci01-j1136640-mgmt-1", component="kubernetes-infrastructure", container="collector", namespace="kube-system", pod="eraser-aks-system-30244119-vmss000000-7fwv2", severity="warning" Description: pod/eraser-aks-system-30244119-vmss000000-7fwv2 in namespace kube-system on container collector has been in waiting state for longer than 1 hour. Firing 7: State: Fired (not resolved) Started: 2026-08-25T12:22:23Z Severity: Sev3 Labels: alertname="KubeContainerWaiting", cluster="ci01-j1136640-mgmt-1", component="kubernetes-infrastructure", container="remover", namespace="kube-system", pod="eraser-aks-system-30244119-vmss000000-7fwv2", severity="warning" Description: pod/eraser-aks-system-30244119-vmss000000-7fwv2 in namespace kube-system on container remover has been in waiting state for longer than 1 hour. Firing 8: State: Fired (not resolved) Started: 2026-08-25T12:22:23Z Severity: Sev3 Labels: alertname="KubeContainerWaiting", cluster="ci01-j1136640-mgmt-1", component="kubernetes-infrastructure", container="node-driver-registrar", namespace="kube-system", pod="aks-secrets-store-csi-driver-2k5cw", severity="warning" Description: pod/aks-secrets-store-csi-driver-2k5cw in namespace kube-system on container node-driver-registrar has been in waiting state for longer than 1 hour.
10:30:20	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 8b8c62c	alert, e2e	failed	#6657 (open)	3	e2e (2) cluster entered terminal Failed state after role assignment deployment 10:30:20 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.002s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.002s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 10:30:20 UTC · failed at e2e · test FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc0009be400>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-m6f64pzjnj2b" to finish creating: GET https://rp.j5727872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/25febb2b-752c-4172-b76b-636cdbb8da75 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/25febb2b-752c-4172-b76b-636cdbb8da75", "name": "25febb2b-752c-4172-b76b-636cdbb8da75", "status": "Failed", "startTime": "2026-08-25T11:26:11.043207919Z", "endTime": "2026-08-25T11:45:13.698680289Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc0009be400>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-m6f64pzjnj2b" to finish creating: GET https://rp.j5727872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/25febb2b-752c-4172-b76b-636cdbb8da75 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/25febb2b-752c-4172-b76b-636cdbb8da75", "name": "25febb2b-752c-4172-b76b-636cdbb8da75", "status": "Failed", "startTime": "2026-08-25T11:26:11.043207919Z", "endTime": "2026-08-25T11:45:13.698680289Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 10:30:20 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-08-25T11:39:31Z Ended: 2026-08-25T12:14:36Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5727872-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdir7piesvgh51a20oka0qaplpe4b0t-x2t4h8s0j2c7z6a", pod="router-c975cc75f-9rwjt", severity="warning" Description: Pod ocm-arohcpci01-2sdir7piesvgh51a20oka0qaplpe4b0t-x2t4h8s0j2c7z6a/router-c975cc75f-9rwjt has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-08-25T11:40:34Z Ended: 2026-08-25T11:58:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5727872-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdirl616dc560s7ergbqgff41rhcqqc-x2z0i1i4i8f5x4u", pod="router-5fbfd4cc9-n5cg7", severity="warning" Description: Pod ocm-arohcpci01-2sdirl616dc560s7ergbqgff41rhcqqc-x2z0i1i4i8f5x4u/router-5fbfd4cc9-n5cg7 has been in a non-ready state for longer than 5 minutes.
10:29:20	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 669f42e	alert, e2e	failed	★#6618 (open)	10	e2e (9) ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift 10:29:20 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-xxw9sj" with version 4.21.29 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000fd07c0>: failed to create HCP cluster cluster-zstream-4-21-xxw9sj: failed waiting for cluster="cluster-zstream-4-21-xxw9sj" in resourcegroup="rg-zstream-upgrade-4-21-c6fglssn9lfl" to finish creating: GET https://rp.j0982912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d2126240-ce19-4a5a-9d90-bf9bef38c389 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d2126240-ce19-4a5a-9d90-bf9bef38c389", "name": "d2126240-ce19-4a5a-9d90-bf9bef38c389", "status": "Failed", "startTime": "2026-08-25T11:13:06.200554482Z", "endTime": "2026-08-25T11:32:15.844191701Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-xxw9sj" with version 4.21.29 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000fd07c0>: failed to create HCP cluster cluster-zstream-4-21-xxw9sj: failed waiting for cluster="cluster-zstream-4-21-xxw9sj" in resourcegroup="rg-zstream-upgrade-4-21-c6fglssn9lfl" to finish creating: GET https://rp.j0982912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d2126240-ce19-4a5a-9d90-bf9bef38c389 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d2126240-ce19-4a5a-9d90-bf9bef38c389", "name": "d2126240-ce19-4a5a-9d90-bf9bef38c389", "status": "Failed", "startTime": "2026-08-25T11:13:06.200554482Z", "endTime": "2026-08-25T11:32:15.844191701Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator]; provider Microsoft.RedHatOpenShift 10:29:20 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0xc000c88360>: failed to create HCP cluster ea-listgrtd5h: failed waiting for cluster="ea-listgrtd5h" in resourcegroup="ea-list-xlnjgvtfpm84" to finish creating: GET https://rp.j0982912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/acc01fc7-72a8-4510-9e4a-d567598b9c1d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/acc01fc7-72a8-4510-9e4a-d567598b9c1d", "name": "acc01fc7-72a8-4510-9e4a-d567598b9c1d", "status": "Failed", "startTime": "2026-08-25T11:13:12.096173667Z", "endTime": "2026-08-25T11:32:15.908926647Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0xc000c88360>: failed to create HCP cluster ea-listgrtd5h: failed waiting for cluster="ea-listgrtd5h" in resourcegroup="ea-list-xlnjgvtfpm84" to finish creating: GET https://rp.j0982912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/acc01fc7-72a8-4510-9e4a-d567598b9c1d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/acc01fc7-72a8-4510-9e4a-d567598b9c1d", "name": "acc01fc7-72a8-4510-9e4a-d567598b9c1d", "status": "Failed", "startTime": "2026-08-25T11:13:12.096173667Z", "endTime": "2026-08-25T11:32:15.908926647Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift 10:29:20 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc000ef9640>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-4hk8t569ldwn" to finish creating: GET https://rp.j0982912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/09ac2de1-d9ce-4cf8-bc9c-7c422d5d09b6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/09ac2de1-d9ce-4cf8-bc9c-7c422d5d09b6", "name": "09ac2de1-d9ce-4cf8-bc9c-7c422d5d09b6", "status": "Failed", "startTime": "2026-08-25T11:13:43.911188079Z", "endTime": "2026-08-25T11:32:45.953478879Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc000ef9640>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-4hk8t569ldwn" to finish creating: GET https://rp.j0982912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/09ac2de1-d9ce-4cf8-bc9c-7c422d5d09b6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/09ac2de1-d9ce-4cf8-bc9c-7c422d5d09b6", "name": "09ac2de1-d9ce-4cf8-bc9c-7c422d5d09b6", "status": "Failed", "startTime": "2026-08-25T11:13:43.911188079Z", "endTime": "2026-08-25T11:32:45.953478879Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift 10:29:20 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc000d6eca0>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-qtr6p2j9mh46" to finish creating: GET https://rp.j0982912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d62ea837-b979-4ca7-86a1-f20df85fd7b1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d62ea837-b979-4ca7-86a1-f20df85fd7b1", "name": "d62ea837-b979-4ca7-86a1-f20df85fd7b1", "status": "Failed", "startTime": "2026-08-25T11:13:16.876555859Z", "endTime": "2026-08-25T11:32:25.915902762Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc000d6eca0>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-qtr6p2j9mh46" to finish creating: GET https://rp.j0982912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d62ea837-b979-4ca7-86a1-f20df85fd7b1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d62ea837-b979-4ca7-86a1-f20df85fd7b1", "name": "d62ea837-b979-4ca7-86a1-f20df85fd7b1", "status": "Failed", "startTime": "2026-08-25T11:13:16.876555859Z", "endTime": "2026-08-25T11:32:25.915902762Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: WaitingForAvailable: Waiting for Kube APIServer deployment to become available; hosted cluster degraded: UnavailableReplicas: [capi-provider, kube-apiserver]; provider Microsoft.RedHatOpenShift 10:29:20 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-t7drrl95cd8z/cluster-candidate-4-20-t2lbd9 should provision Unexpected error: <fmt.wrapError \| 0xc001a36920>: failed to create HCP cluster cluster-candidate-4-20-t2lbd9: failed waiting for cluster="cluster-candidate-4-20-t2lbd9" in resourcegroup="rg-candidate-4-20-t7drrl95cd8z" to finish creating: GET https://rp.j0982912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/579deeb9-f336-4f04-8b2e-b7d680266538 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/579deeb9-f336-4f04-8b2e-b7d680266538", "name": "579deeb9-f336-4f04-8b2e-b7d680266538", "status": "Failed", "startTime": "2026-08-25T11:13:01.56047746Z", "endTime": "2026-08-25T11:32:05.952606176Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: WaitingForAvailable: Waiting for Kube APIServer deployment to become available; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-t7drrl95cd8z/cluster-candidate-4-20-t2lbd9 should provision Unexpected error: <fmt.wrapError \| 0xc001a36920>: failed to create HCP cluster cluster-candidate-4-20-t2lbd9: failed waiting for cluster="cluster-candidate-4-20-t2lbd9" in resourcegroup="rg-candidate-4-20-t7drrl95cd8z" to finish creating: GET https://rp.j0982912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/579deeb9-f336-4f04-8b2e-b7d680266538 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/579deeb9-f336-4f04-8b2e-b7d680266538", "name": "579deeb9-f336-4f04-8b2e-b7d680266538", "status": "Failed", "startTime": "2026-08-25T11:13:01.56047746Z", "endTime": "2026-08-25T11:32:05.952606176Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: WaitingForAvailable: Waiting for Kube APIServer deployment to become available; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:29:20 UTC · failed at e2e · test FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc000978da0>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-zn5xf6rhmm5z" to finish creating: GET https://rp.j0982912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e4881145-71b4-45d4-ac59-0367dec0a789 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e4881145-71b4-45d4-ac59-0367dec0a789", "name": "e4881145-71b4-45d4-ac59-0367dec0a789", "status": "Failed", "startTime": "2026-08-25T11:13:11.121536159Z", "endTime": "2026-08-25T11:32:15.953394704Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc000978da0>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-zn5xf6rhmm5z" to finish creating: GET https://rp.j0982912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e4881145-71b4-45d4-ac59-0367dec0a789 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e4881145-71b4-45d4-ac59-0367dec0a789", "name": "e4881145-71b4-45d4-ac59-0367dec0a789", "status": "Failed", "startTime": "2026-08-25T11:13:11.121536159Z", "endTime": "2026-08-25T11:32:15.953394704Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:29:20 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-k54h9d" with version 4.22.10 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000f9cf00>: failed to create HCP cluster cluster-zstream-4-22-k54h9d: failed waiting for cluster="cluster-zstream-4-22-k54h9d" in resourcegroup="rg-zstream-upgrade-4-22-r5wjf2g99fnb" to finish creating: GET https://rp.j0982912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9fe8d963-e967-44a6-bfa1-19ebb4ebffbc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9fe8d963-e967-44a6-bfa1-19ebb4ebffbc", "name": "9fe8d963-e967-44a6-bfa1-19ebb4ebffbc", "status": "Failed", "startTime": "2026-08-25T11:12:56.985755901Z", "endTime": "2026-08-25T11:32:05.9413244Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-k54h9d" with version 4.22.10 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000f9cf00>: failed to create HCP cluster cluster-zstream-4-22-k54h9d: failed waiting for cluster="cluster-zstream-4-22-k54h9d" in resourcegroup="rg-zstream-upgrade-4-22-r5wjf2g99fnb" to finish creating: GET https://rp.j0982912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9fe8d963-e967-44a6-bfa1-19ebb4ebffbc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9fe8d963-e967-44a6-bfa1-19ebb4ebffbc", "name": "9fe8d963-e967-44a6-bfa1-19ebb4ebffbc", "status": "Failed", "startTime": "2026-08-25T11:12:56.985755901Z", "endTime": "2026-08-25T11:32:05.9413244Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator, ignition-server-proxy, kube-controller-manager]; provider Microsoft.RedHatOpenShift 10:29:20 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-6nw9hm with version 4.20.35 Unexpected error: <fmt.wrapError \| 0xc000892700>: failed to create HCP cluster np-version-upgrade-cluster-6nw9hm: failed waiting for cluster="np-version-upgrade-cluster-6nw9hm" in resourcegroup="rg-np-version-upgrade-6nw9hm-sbc4vzg69snt" to finish creating: GET https://rp.j0982912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dd8368f0-8487-43cb-9ffb-d72a8e31a349 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dd8368f0-8487-43cb-9ffb-d72a8e31a349", "name": "dd8368f0-8487-43cb-9ffb-d72a8e31a349", "status": "Failed", "startTime": "2026-08-25T11:13:18.056611915Z", "endTime": "2026-08-25T11:32:25.82846965Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 2 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-6nw9hm with version 4.20.35 Unexpected error: <fmt.wrapError \| 0xc000892700>: failed to create HCP cluster np-version-upgrade-cluster-6nw9hm: failed waiting for cluster="np-version-upgrade-cluster-6nw9hm" in resourcegroup="rg-np-version-upgrade-6nw9hm-sbc4vzg69snt" to finish creating: GET https://rp.j0982912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dd8368f0-8487-43cb-9ffb-d72a8e31a349 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dd8368f0-8487-43cb-9ffb-d72a8e31a349", "name": "dd8368f0-8487-43cb-9ffb-d72a8e31a349", "status": "Failed", "startTime": "2026-08-25T11:13:18.056611915Z", "endTime": "2026-08-25T11:32:25.82846965Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 2 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:29:20 UTC · failed at e2e · test SRE can pause schedules to stop backup execution for an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc0009101c0>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-vzm85c2qbsgd" to finish creating: GET https://rp.j0982912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/01cf43dd-3011-4a8d-9caf-ea328f7d5fc4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/01cf43dd-3011-4a8d-9caf-ea328f7d5fc4", "name": "01cf43dd-3011-4a8d-9caf-ea328f7d5fc4", "status": "Failed", "startTime": "2026-08-25T11:13:23.203350811Z", "endTime": "2026-08-25T11:32:25.896029624Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc0009101c0>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-vzm85c2qbsgd" to finish creating: GET https://rp.j0982912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/01cf43dd-3011-4a8d-9caf-ea328f7d5fc4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/01cf43dd-3011-4a8d-9caf-ea328f7d5fc4", "name": "01cf43dd-3011-4a8d-9caf-ea328f7d5fc4", "status": "Failed", "startTime": "2026-08-25T11:13:23.203350811Z", "endTime": "2026-08-25T11:32:25.896029624Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 10:29:20 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 70 time(s) (20 unknown, 50 known) Firing 1: State: Resolved Started: 2026-08-25T11:22:16Z Ended: 2026-08-25T11:28:16Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0982912-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdilgqdj36sal4q4e9a57hvh6ajgfbp-j3p9v3z4j5r6p2w", pod="etcd-1", severity="warning" Description: Pod ocm-arohcpci01-2sdilgqdj36sal4q4e9a57hvh6ajgfbp-j3p9v3z4j5r6p2w/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-08-25T11:23:15Z Ended: 2026-08-25T11:29:18Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0982912-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdili48rqlgtv5ms3vjppa997f6gctd-ea-listgrtd5h", pod="cluster-api-57f4cccc64-zkpfw", severity="warning" Description: Pod ocm-arohcpci01-2sdili48rqlgtv5ms3vjppa997f6gctd-ea-listgrtd5h/cluster-api-57f4cccc64-zkpfw has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-08-25T11:23:15Z Ended: 2026-08-25T11:29:19Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0982912-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdilggqeajkcm0ivagj6qovcves1q5b-a1f9o5m3y1a2v1u", pod="etcd-1", severity="warning" Description: Pod ocm-arohcpci01-2sdilggqeajkcm0ivagj6qovcves1q5b-a1f9o5m3y1a2v1u/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-08-25T11:23:15Z Ended: 2026-08-25T11:29:19Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0982912-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdilqfi7gmv0s4r7i9nnu81qe3sfhmu-o7a7g0l5g7d0e6d", pod="control-plane-operator-5846985856-dwpdq", severity="warning" Description: Pod ocm-arohcpci01-2sdilqfi7gmv0s4r7i9nnu81qe3sfhmu-o7a7g0l5g7d0e6d/control-plane-operator-5846985856-dwpdq has been in a non-ready state for longer than 5 minutes. Firing 5: State: Resolved Started: 2026-08-25T11:23:15Z Ended: 2026-08-25T11:31:18Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0982912-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdilgko7uh068qmibb2p2nl80hmcr2t-sre-hcp-cluster", pod="kube-apiserver-6fff59d9b9-r7c94", severity="warning" Description: Pod ocm-arohcpci01-2sdilgko7uh068qmibb2p2nl80hmcr2t-sre-hcp-cluster/kube-apiserver-6fff59d9b9-r7c94 has been in a non-ready state for longer than 5 minutes. Firing 6: State: Resolved Started: 2026-08-25T11:23:15Z Ended: 2026-08-25T11:29:19Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0982912-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdiljh6ae8gvdi928aqg7igno5v6m4a-m9l2r5o1n9x5m1r", pod="etcd-1", severity="warning" Description: Pod ocm-arohcpci01-2sdiljh6ae8gvdi928aqg7igno5v6m4a-m9l2r5o1n9x5m1r/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 7: State: Resolved Started: 2026-08-25T11:23:15Z Ended: 2026-08-25T11:29:19Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0982912-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdilimj0peupdgmcd9ptu4jvrh6qg2d-c3m2x0d8u8p4q0n", pod="etcd-0", severity="warning" Description: Pod ocm-arohcpci01-2sdilimj0peupdgmcd9ptu4jvrh6qg2d-c3m2x0d8u8p4q0n/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 8: State: Resolved Started: 2026-08-25T11:23:15Z Ended: 2026-08-25T11:29:19Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0982912-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdile8siqsvbkl4sfirmman2o5ffnof-k8a3l8q9y2b8h3z", pod="etcd-0", severity="warning" Description: Pod ocm-arohcpci01-2sdile8siqsvbkl4sfirmman2o5ffnof-k8a3l8q9y2b8h3z/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 9: State: Resolved Started: 2026-08-25T11:23:15Z Ended: 2026-08-25T11:30:18Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0982912-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdili0tb2m5lpededoul600v6dh93b7-j9q2y1u3s0w4r7x", pod="etcd-0", severity="warning" Description: Pod ocm-arohcpci01-2sdili0tb2m5lpededoul600v6dh93b7-j9q2y1u3s0w4r7x/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 10: State: Resolved Started: 2026-08-25T11:23:15Z Ended: 2026-08-25T11:29:19Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0982912-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdiltvi8g0n5p1koouov5n23ea3c448-g6d6q7a2u2x0l7v", pod="etcd-1", severity="warning" Description: Pod ocm-arohcpci01-2sdiltvi8g0n5p1koouov5n23ea3c448-g6d6q7a2u2x0l7v/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 11: State: Resolved Started: 2026-08-25T11:23:15Z Ended: 2026-08-25T11:29:19Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0982912-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdiltvi8g0n5p1koouov5n23ea3c448-g6d6q7a2u2x0l7v", pod="etcd-0", severity="warning" Description: Pod ocm-arohcpci01-2sdiltvi8g0n5p1koouov5n23ea3c448-g6d6q7a2u2x0l7v/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 12: State: Resolved Started: 2026-08-25T11:23:15Z Ended: 2026-08-25T11:30:19Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0982912-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdilm4kp8ht0oeq4rrf02uregq5naqb-r1e8l8q2p0u8h1y", pod="etcd-2", severity="warning" Description: Pod ocm-arohcpci01-2sdilm4kp8ht0oeq4rrf02uregq5naqb-r1e8l8q2p0u8h1y/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 13: State: Resolved Started: 2026-08-25T11:23:16Z Ended: 2026-08-25T11:29:19Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0982912-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdilqm74gj28677ca8klu0le33fadfb-c6e6h8e6t5t1g7r", pod="etcd-2", severity="warning" Description: Pod ocm-arohcpci01-2sdilqm74gj28677ca8klu0le33fadfb-c6e6h8e6t5t1g7r/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 14: State: Resolved Started: 2026-08-25T11:23:16Z Ended: 2026-08-25T11:29:19Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0982912-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdilf9lrljchocg563vtvvjb0a24i8h-i5t4p8s6m7u7k8a", pod="etcd-0", severity="warning" Description: Pod ocm-arohcpci01-2sdilf9lrljchocg563vtvvjb0a24i8h-i5t4p8s6m7u7k8a/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 15: State: Resolved Started: 2026-08-25T11:24:14Z Ended: 2026-08-25T11:30:17Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0982912-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdilkvrcjt48hulqqgsqhblsbqojees-p0x0o6d8w2v9t8p", pod="etcd-1", severity="warning" Description: Pod ocm-arohcpci01-2sdilkvrcjt48hulqqgsqhblsbqojees-p0x0o6d8w2v9t8p/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 16: State: Resolved Started: 2026-08-25T11:24:14Z Ended: 2026-08-25T11:30:17Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0982912-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdili0ibn99qtvs4jjc0tppu5v61opj-d3v7t2v2y2v6h1k", pod="etcd-2", severity="warning" Description: Pod ocm-arohcpci01-2sdili0ibn99qtvs4jjc0tppu5v61opj-d3v7t2v2y2v6h1k/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 17: State: Resolved Started: 2026-08-25T11:24:14Z Ended: 2026-08-25T11:30:19Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0982912-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdilj8q9l3vpagobot4o90tggf7475q-b0e5d6r8d2f9p5i", pod="etcd-1", severity="warning" Description: Pod ocm-arohcpci01-2sdilj8q9l3vpagobot4o90tggf7475q-b0e5d6r8d2f9p5i/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 18: State: Resolved Started: 2026-08-25T11:24:14Z Ended: 2026-08-25T11:30:19Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0982912-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdilgq8asvq19tc9m6el9uvbgv8ngs6-g7b1y4g7p8f2l1r", pod="etcd-1", severity="warning" Description: Pod ocm-arohcpci01-2sdilgq8asvq19tc9m6el9uvbgv8ngs6-g7b1y4g7p8f2l1r/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 19: State: Resolved Started: 2026-08-25T11:25:15Z Ended: 2026-08-25T11:31:19Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0982912-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdilegalafbqc4itqpt0up25tbij7h7-i7l0h0h4g1d6r4d", pod="kube-apiserver-75fcd8657-xclt4", severity="warning" Description: Pod ocm-arohcpci01-2sdilegalafbqc4itqpt0up25tbij7h7-i7l0h0h4g1d6r4d/kube-apiserver-75fcd8657-xclt4 has been in a non-ready state for longer than 5 minutes. Firing 20: State: Resolved Started: 2026-08-25T11:25:15Z Ended: 2026-08-25T11:31:19Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0982912-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdile6vap1e17rrbb0clqfaohcoisog-u2k1x5w3i5n2n6p", pod="kube-apiserver-6f5cf68c44-rbm77", severity="warning" Description: Pod ocm-arohcpci01-2sdile6vap1e17rrbb0clqfaohcoisog-u2k1x5w3i5n2n6p/kube-apiserver-6f5cf68c44-rbm77 has been in a non-ready state for longer than 5 minutes.
10:22:57	pull-ci-Azure-ARO-HCP-main-e2e-parallel head b89541a	alert, e2e	failed	⚠#6670 (open)	29	e2e (27) ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: WaitingForAvailable: Waiting for Kube APIServer deployment to become available; hosted cluster degraded: UnavailableReplicas: [capi-provider, kube-apiserver]; provider Microsoft.RedHatOpenShift 10:22:57 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-zhhnpcphpl7r/cluster-candidate-4-21-tcc8ch should provision Unexpected error: <fmt.wrapError \| 0xc000b2aa60>: failed to create HCP cluster cluster-candidate-4-21-tcc8ch: failed waiting for cluster="cluster-candidate-4-21-tcc8ch" in resourcegroup="rg-candidate-4-21-zhhnpcphpl7r" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/06c23c7d-cba1-433b-881e-637949594df3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/06c23c7d-cba1-433b-881e-637949594df3", "name": "06c23c7d-cba1-433b-881e-637949594df3", "status": "Failed", "startTime": "2026-08-25T11:01:29.520227726Z", "endTime": "2026-08-25T11:20:37.842143923Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: WaitingForAvailable: Waiting for Kube APIServer deployment to become available; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-zhhnpcphpl7r/cluster-candidate-4-21-tcc8ch should provision Unexpected error: <fmt.wrapError \| 0xc000b2aa60>: failed to create HCP cluster cluster-candidate-4-21-tcc8ch: failed waiting for cluster="cluster-candidate-4-21-tcc8ch" in resourcegroup="rg-candidate-4-21-zhhnpcphpl7r" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/06c23c7d-cba1-433b-881e-637949594df3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/06c23c7d-cba1-433b-881e-637949594df3", "name": "06c23c7d-cba1-433b-881e-637949594df3", "status": "Failed", "startTime": "2026-08-25T11:01:29.520227726Z", "endTime": "2026-08-25T11:20:37.842143923Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: WaitingForAvailable: Waiting for Kube APIServer deployment to become available; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 10:22:57 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc001076940>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-zlx9rbktl8hf" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/12f586e0-5a79-48d8-98cc-d9683f3d6329 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/12f586e0-5a79-48d8-98cc-d9683f3d6329", "name": "12f586e0-5a79-48d8-98cc-d9683f3d6329", "status": "Failed", "startTime": "2026-08-25T11:01:39.436649403Z", "endTime": "2026-08-25T11:20:48.141412603Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, dns-operator, community-operators-catalog, azure-cloud-controller-manager, ingress-operator, csi-snapshot-controller-operator, redhat-operators-catalog, cluster-version-operator, cluster-storage-operator, redhat-marketplace-catalog, catalog-operator, olm-operator, hosted-cluster-config-operator, packageserver, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc001076940>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-zlx9rbktl8hf" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/12f586e0-5a79-48d8-98cc-d9683f3d6329 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/12f586e0-5a79-48d8-98cc-d9683f3d6329", "name": "12f586e0-5a79-48d8-98cc-d9683f3d6329", "status": "Failed", "startTime": "2026-08-25T11:01:39.436649403Z", "endTime": "2026-08-25T11:20:48.141412603Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, dns-operator, community-operators-catalog, azure-cloud-controller-manager, ingress-operator, csi-snapshot-controller-operator, redhat-operators-catalog, cluster-version-operator, cluster-storage-operator, redhat-marketplace-catalog, catalog-operator, olm-operator, hosted-cluster-config-operator, packageserver, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-controller-manager]; provider Microsoft.RedHatOpenShift 10:22:57 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:721]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc00069e2c0>: failed to create HCP cluster np-dg-minor-zd5jtq: failed waiting for cluster="np-dg-minor-zd5jtq" in resourcegroup="rg-np-dg-minor-zd5jtq-kkl7vl8r2p4z" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/20b0f9df-28c2-45ff-b6bf-ca10517cb3f4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/20b0f9df-28c2-45ff-b6bf-ca10517cb3f4", "name": "20b0f9df-28c2-45ff-b6bf-ca10517cb3f4", "status": "Failed", "startTime": "2026-08-25T11:01:30.680263097Z", "endTime": "2026-08-25T11:20:37.924037446Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 2 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:721]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc00069e2c0>: failed to create HCP cluster np-dg-minor-zd5jtq: failed waiting for cluster="np-dg-minor-zd5jtq" in resourcegroup="rg-np-dg-minor-zd5jtq-kkl7vl8r2p4z" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/20b0f9df-28c2-45ff-b6bf-ca10517cb3f4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/20b0f9df-28c2-45ff-b6bf-ca10517cb3f4", "name": "20b0f9df-28c2-45ff-b6bf-ca10517cb3f4", "status": "Failed", "startTime": "2026-08-25T11:01:30.680263097Z", "endTime": "2026-08-25T11:20:37.924037446Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 2 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 10:22:57 UTC · failed at e2e · test SRE can pause schedules to stop backup execution for an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000889040>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-794c9cl949m2" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8741a29d-3a29-4b25-9cca-bfb6b98db145 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8741a29d-3a29-4b25-9cca-bfb6b98db145", "name": "8741a29d-3a29-4b25-9cca-bfb6b98db145", "status": "Failed", "startTime": "2026-08-25T11:01:20.170201007Z", "endTime": "2026-08-25T11:20:27.872895981Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000889040>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-794c9cl949m2" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8741a29d-3a29-4b25-9cca-bfb6b98db145 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8741a29d-3a29-4b25-9cca-bfb6b98db145", "name": "8741a29d-3a29-4b25-9cca-bfb6b98db145", "status": "Failed", "startTime": "2026-08-25T11:01:20.170201007Z", "endTime": "2026-08-25T11:20:27.872895981Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 10:22:57 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:122]: failed to create nodepool ephemeral-np with ephemeral OS disk Unexpected error: <fmt.wrapError \| 0xc000943300>: failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-4hldwq4wn65q" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7b775fba-4326-448d-9ed3-d84953f5f41e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7b775fba-4326-448d-9ed3-d84953f5f41e", "name": "7b775fba-4326-448d-9ed3-d84953f5f41e", "status": "Failed", "startTime": "2026-08-25T11:18:57.616565456Z", "endTime": "2026-08-25T11:38:04.521702463Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:122]: failed to create nodepool ephemeral-np with ephemeral OS disk Unexpected error: <fmt.wrapError \| 0xc000943300>: failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-4hldwq4wn65q" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7b775fba-4326-448d-9ed3-d84953f5f41e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7b775fba-4326-448d-9ed3-d84953f5f41e", "name": "7b775fba-4326-448d-9ed3-d84953f5f41e", "status": "Failed", "startTime": "2026-08-25T11:18:57.616565456Z", "endTime": "2026-08-25T11:38:04.521702463Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:22:57 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0xc0010427e0>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-scdxnxk455c9" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eb03150b-7aa4-4977-bdd5-3bf252c14ed8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eb03150b-7aa4-4977-bdd5-3bf252c14ed8", "name": "eb03150b-7aa4-4977-bdd5-3bf252c14ed8", "status": "Failed", "startTime": "2026-08-25T11:01:30.520597523Z", "endTime": "2026-08-25T11:20:37.903541851Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0xc0010427e0>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-scdxnxk455c9" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eb03150b-7aa4-4977-bdd5-3bf252c14ed8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eb03150b-7aa4-4977-bdd5-3bf252c14ed8", "name": "eb03150b-7aa4-4977-bdd5-3bf252c14ed8", "status": "Failed", "startTime": "2026-08-25T11:01:30.520597523Z", "endTime": "2026-08-25T11:20:37.903541851Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator, ignition-server-proxy]; provider Microsoft.RedHatOpenShift 10:22:57 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '20.000000' minutes) Unexpected error: <exported.ResponseError \| 0xc0011325d0>: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/642ab867-7117-49d1-9205-fcc81b8fcac8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/642ab867-7117-49d1-9205-fcc81b8fcac8", "name": "642ab867-7117-49d1-9205-fcc81b8fcac8", "status": "Failed", "startTime": "2026-08-25T11:03:23.530391484Z", "endTime": "2026-08-25T11:22:27.900506106Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '20.000000' minutes) Unexpected error: <exported.ResponseError \| 0xc0011325d0>: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/642ab867-7117-49d1-9205-fcc81b8fcac8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/642ab867-7117-49d1-9205-fcc81b8fcac8", "name": "642ab867-7117-49d1-9205-fcc81b8fcac8", "status": "Failed", "startTime": "2026-08-25T11:03:23.530391484Z", "endTime": "2026-08-25T11:22:27.900506106Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift 10:22:57 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260630preview and verify the ingress is internal · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc000b4eac0>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-dvdtjf96rsnl" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bf5319d7-d2c8-4242-876b-f8a97dfb3996 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bf5319d7-d2c8-4242-876b-f8a97dfb3996", "name": "bf5319d7-d2c8-4242-876b-f8a97dfb3996", "status": "Failed", "startTime": "2026-08-25T11:01:39.266118595Z", "endTime": "2026-08-25T11:20:48.041047422Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc000b4eac0>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-dvdtjf96rsnl" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bf5319d7-d2c8-4242-876b-f8a97dfb3996 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bf5319d7-d2c8-4242-876b-f8a97dfb3996", "name": "bf5319d7-d2c8-4242-876b-f8a97dfb3996", "status": "Failed", "startTime": "2026-08-25T11:01:39.266118595Z", "endTime": "2026-08-25T11:20:48.041047422Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:22:57 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-j5vbtp" with version 4.20.34 on candidate channel Unexpected error: <fmt.wrapError \| 0xc00119a640>: failed to create HCP cluster cluster-zstream-4-20-j5vbtp: failed waiting for cluster="cluster-zstream-4-20-j5vbtp" in resourcegroup="rg-zstream-upgrade-4-20-xln7ltx8rz8n" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/141fab79-05e7-48e2-a785-31f4588cc106 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/141fab79-05e7-48e2-a785-31f4588cc106", "name": "141fab79-05e7-48e2-a785-31f4588cc106", "status": "Failed", "startTime": "2026-08-25T11:01:30.143019098Z", "endTime": "2026-08-25T11:20:37.940391987Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-j5vbtp" with version 4.20.34 on candidate channel Unexpected error: <fmt.wrapError \| 0xc00119a640>: failed to create HCP cluster cluster-zstream-4-20-j5vbtp: failed waiting for cluster="cluster-zstream-4-20-j5vbtp" in resourcegroup="rg-zstream-upgrade-4-20-xln7ltx8rz8n" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/141fab79-05e7-48e2-a785-31f4588cc106 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/141fab79-05e7-48e2-a785-31f4588cc106", "name": "141fab79-05e7-48e2-a785-31f4588cc106", "status": "Failed", "startTime": "2026-08-25T11:01:30.143019098Z", "endTime": "2026-08-25T11:20:37.940391987Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 10:22:57 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling Unexpected error: <fmt.wrapError \| 0xc000bc9280>: failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-ldt9sxjdns5j" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/207b6e6b-d81f-4ae8-a52d-235354977a78 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/207b6e6b-d81f-4ae8-a52d-235354977a78", "name": "207b6e6b-d81f-4ae8-a52d-235354977a78", "status": "Failed", "startTime": "2026-08-25T11:20:21.066955417Z", "endTime": "2026-08-25T11:39:24.495248134Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling Unexpected error: <fmt.wrapError \| 0xc000bc9280>: failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-ldt9sxjdns5j" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/207b6e6b-d81f-4ae8-a52d-235354977a78 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/207b6e6b-d81f-4ae8-a52d-235354977a78", "name": "207b6e6b-d81f-4ae8-a52d-235354977a78", "status": "Failed", "startTime": "2026-08-25T11:20:21.066955417Z", "endTime": "2026-08-25T11:39:24.495248134Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-controller-manager]; provider Microsoft.RedHatOpenShift 10:22:57 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc000a84540>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-qsmsgbz4r9bl" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eb200329-6a1f-4203-89e8-8bd77fdd326b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eb200329-6a1f-4203-89e8-8bd77fdd326b", "name": "eb200329-6a1f-4203-89e8-8bd77fdd326b", "status": "Failed", "startTime": "2026-08-25T11:03:10.235142906Z", "endTime": "2026-08-25T11:22:17.989968047Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc000a84540>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-qsmsgbz4r9bl" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eb200329-6a1f-4203-89e8-8bd77fdd326b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eb200329-6a1f-4203-89e8-8bd77fdd326b", "name": "eb200329-6a1f-4203-89e8-8bd77fdd326b", "status": "Failed", "startTime": "2026-08-25T11:03:10.235142906Z", "endTime": "2026-08-25T11:22:17.989968047Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-controller-manager]; provider Microsoft.RedHatOpenShift 10:22:57 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0xc000b8c1c0>: failed to create HCP cluster ea-listwl9qrq: failed waiting for cluster="ea-listwl9qrq" in resourcegroup="ea-list-cs4nbh2vqlnx" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ee1124f8-4c79-44f0-8dc1-bd5636fec70e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ee1124f8-4c79-44f0-8dc1-bd5636fec70e", "name": "ee1124f8-4c79-44f0-8dc1-bd5636fec70e", "status": "Failed", "startTime": "2026-08-25T11:01:19.900770335Z", "endTime": "2026-08-25T11:20:27.863710783Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0xc000b8c1c0>: failed to create HCP cluster ea-listwl9qrq: failed waiting for cluster="ea-listwl9qrq" in resourcegroup="ea-list-cs4nbh2vqlnx" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ee1124f8-4c79-44f0-8dc1-bd5636fec70e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ee1124f8-4c79-44f0-8dc1-bd5636fec70e", "name": "ee1124f8-4c79-44f0-8dc1-bd5636fec70e", "status": "Failed", "startTime": "2026-08-25T11:01:19.900770335Z", "endTime": "2026-08-25T11:20:27.863710783Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 10:22:57 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster Unexpected error: <fmt.wrapError \| 0xc000338280>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-jjh86gpxzmk5" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f4412341-97ae-4d89-b70d-d190163e6b84 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f4412341-97ae-4d89-b70d-d190163e6b84", "name": "f4412341-97ae-4d89-b70d-d190163e6b84", "status": "Failed", "startTime": "2026-08-25T11:14:26.528402468Z", "endTime": "2026-08-25T11:33:34.597854788Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster Unexpected error: <fmt.wrapError \| 0xc000338280>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-jjh86gpxzmk5" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f4412341-97ae-4d89-b70d-d190163e6b84 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f4412341-97ae-4d89-b70d-d190163e6b84", "name": "f4412341-97ae-4d89-b70d-d190163e6b84", "status": "Failed", "startTime": "2026-08-25T11:14:26.528402468Z", "endTime": "2026-08-25T11:33:34.597854788Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift 10:22:57 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-nmv2vp with version 4.20.35 Unexpected error: <fmt.wrapError \| 0xc00115a280>: failed to create HCP cluster np-version-upgrade-cluster-nmv2vp: failed waiting for cluster="np-version-upgrade-cluster-nmv2vp" in resourcegroup="rg-np-version-upgrade-nmv2vp-8jkn6dmxnnxc" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/309d309a-4846-475e-aa8f-a51a3a3cc734 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/309d309a-4846-475e-aa8f-a51a3a3cc734", "name": "309d309a-4846-475e-aa8f-a51a3a3cc734", "status": "Failed", "startTime": "2026-08-25T11:01:41.544397084Z", "endTime": "2026-08-25T11:20:48.26115411Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-nmv2vp with version 4.20.35 Unexpected error: <fmt.wrapError \| 0xc00115a280>: failed to create HCP cluster np-version-upgrade-cluster-nmv2vp: failed waiting for cluster="np-version-upgrade-cluster-nmv2vp" in resourcegroup="rg-np-version-upgrade-nmv2vp-8jkn6dmxnnxc" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/309d309a-4846-475e-aa8f-a51a3a3cc734 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/309d309a-4846-475e-aa8f-a51a3a3cc734", "name": "309d309a-4846-475e-aa8f-a51a3a3cc734", "status": "Failed", "startTime": "2026-08-25T11:01:41.544397084Z", "endTime": "2026-08-25T11:20:48.26115411Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 10:22:57 UTC · failed at e2e · test FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc000a20320>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-jxtcfnn8kwsp" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/42062015-dca9-4bc0-aad5-ab4a1b9d09de -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/42062015-dca9-4bc0-aad5-ab4a1b9d09de", "name": "42062015-dca9-4bc0-aad5-ab4a1b9d09de", "status": "Failed", "startTime": "2026-08-25T11:01:29.494264153Z", "endTime": "2026-08-25T11:20:37.948177471Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, ingress-operator, packageserver, redhat-operators-catalog, community-operators-catalog, cluster-storage-operator, cluster-network-operator, olm-operator, certified-operators-catalog, dns-operator, catalog-operator, redhat-marketplace-catalog, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc000a20320>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-jxtcfnn8kwsp" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/42062015-dca9-4bc0-aad5-ab4a1b9d09de -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/42062015-dca9-4bc0-aad5-ab4a1b9d09de", "name": "42062015-dca9-4bc0-aad5-ab4a1b9d09de", "status": "Failed", "startTime": "2026-08-25T11:01:29.494264153Z", "endTime": "2026-08-25T11:20:37.948177471Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, ingress-operator, packageserver, redhat-operators-catalog, community-operators-catalog, cluster-storage-operator, cluster-network-operator, olm-operator, certified-operators-catalog, dns-operator, catalog-operator, redhat-marketplace-catalog, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 10:22:57 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc000ae8400>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-264hwln96qbw" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f2ee6a72-aa37-4fa5-8b5f-6440eb7438e4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f2ee6a72-aa37-4fa5-8b5f-6440eb7438e4", "name": "f2ee6a72-aa37-4fa5-8b5f-6440eb7438e4", "status": "Failed", "startTime": "2026-08-25T11:01:40.596159853Z", "endTime": "2026-08-25T11:20:48.273802602Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, redhat-operators-catalog, certified-operators-catalog, catalog-operator, community-operators-catalog, ingress-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc000ae8400>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-264hwln96qbw" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f2ee6a72-aa37-4fa5-8b5f-6440eb7438e4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f2ee6a72-aa37-4fa5-8b5f-6440eb7438e4", "name": "f2ee6a72-aa37-4fa5-8b5f-6440eb7438e4", "status": "Failed", "startTime": "2026-08-25T11:01:40.596159853Z", "endTime": "2026-08-25T11:20:48.273802602Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, redhat-operators-catalog, certified-operators-catalog, catalog-operator, community-operators-catalog, ingress-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, machine-approver, olm-operator, openshift-apiserver, packageserver, router]; provider Microsoft.RedHatOpenShift 10:22:57 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--2zj5qrnqfwdn/cluster-hs-pre-4n4p2k should provision Unexpected error: <fmt.wrapError \| 0xc0007e89c0>: failed to create HCP cluster cluster-hs-pre-4n4p2k: failed waiting for cluster="cluster-hs-pre-4n4p2k" in resourcegroup="rg-hs-pre--2zj5qrnqfwdn" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7939f0c1-73a2-48bf-80a1-c5a81f59060f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7939f0c1-73a2-48bf-80a1-c5a81f59060f", "name": "7939f0c1-73a2-48bf-80a1-c5a81f59060f", "status": "Failed", "startTime": "2026-08-25T11:01:28.656859519Z", "endTime": "2026-08-25T11:20:37.93508057Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, cluster-policy-controller, cluster-storage-operator, cluster-node-tuning-operator, dns-operator, cluster-image-registry-operator, packageserver, certified-operators-catalog, hosted-cluster-config-operator, olm-operator, openshift-route-controller-manager, openshift-controller-manager, catalog-operator, cluster-network-operator, machine-approver, csi-snapshot-controller-operator, community-operators-catalog, ingress-operator, redhat-operators-catalog, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--2zj5qrnqfwdn/cluster-hs-pre-4n4p2k should provision Unexpected error: <fmt.wrapError \| 0xc0007e89c0>: failed to create HCP cluster cluster-hs-pre-4n4p2k: failed waiting for cluster="cluster-hs-pre-4n4p2k" in resourcegroup="rg-hs-pre--2zj5qrnqfwdn" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7939f0c1-73a2-48bf-80a1-c5a81f59060f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7939f0c1-73a2-48bf-80a1-c5a81f59060f", "name": "7939f0c1-73a2-48bf-80a1-c5a81f59060f", "status": "Failed", "startTime": "2026-08-25T11:01:28.656859519Z", "endTime": "2026-08-25T11:20:37.93508057Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, cluster-policy-controller, cluster-storage-operator, cluster-node-tuning-operator, dns-operator, cluster-image-registry-operator, packageserver, certified-operators-catalog, hosted-cluster-config-operator, olm-operator, openshift-route-controller-manager, openshift-controller-manager, catalog-operator, cluster-network-operator, machine-approver, csi-snapshot-controller-operator, community-operators-catalog, ingress-operator, redhat-operators-catalog, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift 10:22:57 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test Unexpected error: <fmt.wrapError \| 0xc000740280>: failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-66vggwmcrfkm" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ed9a849-4e1a-47be-9fbf-9892b4e55909 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ed9a849-4e1a-47be-9fbf-9892b4e55909", "name": "5ed9a849-4e1a-47be-9fbf-9892b4e55909", "status": "Failed", "startTime": "2026-08-25T11:00:58.915429667Z", "endTime": "2026-08-25T11:20:07.870969411Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test Unexpected error: <fmt.wrapError \| 0xc000740280>: failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-66vggwmcrfkm" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ed9a849-4e1a-47be-9fbf-9892b4e55909 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ed9a849-4e1a-47be-9fbf-9892b4e55909", "name": "5ed9a849-4e1a-47be-9fbf-9892b4e55909", "status": "Failed", "startTime": "2026-08-25T11:00:58.915429667Z", "endTime": "2026-08-25T11:20:07.870969411Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:22:57 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128" Unexpected error: <fmt.wrapError \| 0xc000fac240>: failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-k5wdpllgjssx" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8a6c32ee-264c-4461-998b-e5b7a3d5b8ad -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8a6c32ee-264c-4461-998b-e5b7a3d5b8ad", "name": "8a6c32ee-264c-4461-998b-e5b7a3d5b8ad", "status": "Failed", "startTime": "2026-08-25T11:01:40.221887867Z", "endTime": "2026-08-25T11:20:48.105155011Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128" Unexpected error: <fmt.wrapError \| 0xc000fac240>: failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-k5wdpllgjssx" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8a6c32ee-264c-4461-998b-e5b7a3d5b8ad -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8a6c32ee-264c-4461-998b-e5b7a3d5b8ad", "name": "8a6c32ee-264c-4461-998b-e5b7a3d5b8ad", "status": "Failed", "startTime": "2026-08-25T11:01:40.221887867Z", "endTime": "2026-08-25T11:20:48.105155011Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator]; provider Microsoft.RedHatOpenShift 10:22:57 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000c409e0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-g7htcsxqgjh8" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dffc5a1b-8a6e-474f-9d2b-542515a5266b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dffc5a1b-8a6e-474f-9d2b-542515a5266b", "name": "dffc5a1b-8a6e-474f-9d2b-542515a5266b", "status": "Failed", "startTime": "2026-08-25T11:04:51.512570533Z", "endTime": "2026-08-25T11:23:58.061151727Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator; hosted cluster degraded: UnavailableReplicas: cluster-network-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000c409e0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-g7htcsxqgjh8" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dffc5a1b-8a6e-474f-9d2b-542515a5266b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dffc5a1b-8a6e-474f-9d2b-542515a5266b", "name": "dffc5a1b-8a6e-474f-9d2b-542515a5266b", "status": "Failed", "startTime": "2026-08-25T11:04:51.512570533Z", "endTime": "2026-08-25T11:23:58.061151727Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator; hosted cluster degraded: UnavailableReplicas: cluster-network-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 10:22:57 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc0010626e0>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-6xkgnbgv84t9" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f7966876-e222-4b01-94a7-f89b2c83e042 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f7966876-e222-4b01-94a7-f89b2c83e042", "name": "f7966876-e222-4b01-94a7-f89b2c83e042", "status": "Failed", "startTime": "2026-08-25T11:04:02.696486003Z", "endTime": "2026-08-25T11:23:08.046324509Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, dns-operator, csi-snapshot-controller-operator, packageserver, catalog-operator, cluster-version-operator, olm-operator, cluster-network-operator, hosted-cluster-config-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 2 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc0010626e0>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-6xkgnbgv84t9" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f7966876-e222-4b01-94a7-f89b2c83e042 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f7966876-e222-4b01-94a7-f89b2c83e042", "name": "f7966876-e222-4b01-94a7-f89b2c83e042", "status": "Failed", "startTime": "2026-08-25T11:04:02.696486003Z", "endTime": "2026-08-25T11:23:08.046324509Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, dns-operator, csi-snapshot-controller-operator, packageserver, catalog-operator, cluster-version-operator, olm-operator, cluster-network-operator, hosted-cluster-config-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 2 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 10:22:57 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-f9drnxf8872v/cluster-candidate-4-20-7gm7rd should provision Unexpected error: <fmt.wrapError \| 0xc001b08b00>: failed to create HCP cluster cluster-candidate-4-20-7gm7rd: failed waiting for cluster="cluster-candidate-4-20-7gm7rd" in resourcegroup="rg-candidate-4-20-f9drnxf8872v" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c0d278c3-b1dc-43c0-9907-4e3f268841ce -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c0d278c3-b1dc-43c0-9907-4e3f268841ce", "name": "c0d278c3-b1dc-43c0-9907-4e3f268841ce", "status": "Failed", "startTime": "2026-08-25T11:01:29.794018971Z", "endTime": "2026-08-25T11:20:37.949376228Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, ingress-operator, openshift-controller-manager, dns-operator, csi-snapshot-controller-operator, olm-operator, cluster-version-operator, packageserver, cluster-node-tuning-operator, hosted-cluster-config-operator, openshift-route-controller-manager, cluster-network-operator, cluster-image-registry-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 2 unavailable replicas, openshift-route-controller-manager deployment has 2 unavailable replicas, packageserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-f9drnxf8872v/cluster-candidate-4-20-7gm7rd should provision Unexpected error: <fmt.wrapError \| 0xc001b08b00>: failed to create HCP cluster cluster-candidate-4-20-7gm7rd: failed waiting for cluster="cluster-candidate-4-20-7gm7rd" in resourcegroup="rg-candidate-4-20-f9drnxf8872v" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c0d278c3-b1dc-43c0-9907-4e3f268841ce -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c0d278c3-b1dc-43c0-9907-4e3f268841ce", "name": "c0d278c3-b1dc-43c0-9907-4e3f268841ce", "status": "Failed", "startTime": "2026-08-25T11:01:29.794018971Z", "endTime": "2026-08-25T11:20:37.949376228Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, ingress-operator, openshift-controller-manager, dns-operator, csi-snapshot-controller-operator, olm-operator, cluster-version-operator, packageserver, cluster-node-tuning-operator, hosted-cluster-config-operator, openshift-route-controller-manager, cluster-network-operator, cluster-image-registry-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 2 unavailable replicas, openshift-route-controller-manager deployment has 2 unavailable replicas, packageserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager, catalog-operator, hosted-cluster-config-operator, kube-storage-version-migrator, machine-approver, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 10:22:57 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-p7lpzhts95vd/cluster-candidate-4-22-hlfnj9 should provision Unexpected error: <fmt.wrapError \| 0xc0005ad040>: failed to create HCP cluster cluster-candidate-4-22-hlfnj9: failed waiting for cluster="cluster-candidate-4-22-hlfnj9" in resourcegroup="rg-candidate-4-22-p7lpzhts95vd" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/18015e61-c049-41a2-809f-ad31666a4af5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/18015e61-c049-41a2-809f-ad31666a4af5", "name": "18015e61-c049-41a2-809f-ad31666a4af5", "status": "Failed", "startTime": "2026-08-25T11:01:40.338382698Z", "endTime": "2026-08-25T11:20:48.26963606Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, cluster-image-registry-operator, ingress-operator, kube-storage-version-migrator, catalog-operator, packageserver, certified-operators-catalog, csi-snapshot-controller-operator, community-operators-catalog, cluster-version-operator, azure-cloud-controller-manager, cluster-storage-operator, redhat-operators-catalog, cluster-policy-controller, cluster-network-operator, machine-approver, dns-operator, olm-operator, openshift-apiserver, openshift-route-controller-manager, hosted-cluster-config-operator, cluster-node-tuning-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 3 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-p7lpzhts95vd/cluster-candidate-4-22-hlfnj9 should provision Unexpected error: <fmt.wrapError \| 0xc0005ad040>: failed to create HCP cluster cluster-candidate-4-22-hlfnj9: failed waiting for cluster="cluster-candidate-4-22-hlfnj9" in resourcegroup="rg-candidate-4-22-p7lpzhts95vd" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/18015e61-c049-41a2-809f-ad31666a4af5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/18015e61-c049-41a2-809f-ad31666a4af5", "name": "18015e61-c049-41a2-809f-ad31666a4af5", "status": "Failed", "startTime": "2026-08-25T11:01:40.338382698Z", "endTime": "2026-08-25T11:20:48.26963606Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, cluster-image-registry-operator, ingress-operator, kube-storage-version-migrator, catalog-operator, packageserver, certified-operators-catalog, csi-snapshot-controller-operator, community-operators-catalog, cluster-version-operator, azure-cloud-controller-manager, cluster-storage-operator, redhat-operators-catalog, cluster-policy-controller, cluster-network-operator, machine-approver, dns-operator, olm-operator, openshift-apiserver, openshift-route-controller-manager, hosted-cluster-config-operator, cluster-node-tuning-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 3 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 10:22:57 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-nkwvwj" for cluster "cluster-candidate-5-0-nkwvwj" Unexpected error: <fmt.wrapError \| 0xc000179060>: failed to create NodePool np-nkwvwj: failed waiting for nodepool="np-nkwvwj" for cluster "cluster-candidate-5-0-nkwvwj" in resourcegroup="rg-candidate-5-0-qh5dlq8h25tn" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/02a0680c-d02a-4979-935f-a2788b9917ce -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/02a0680c-d02a-4979-935f-a2788b9917ce", "name": "02a0680c-d02a-4979-935f-a2788b9917ce", "status": "Failed", "startTime": "2026-08-25T11:20:40.33393292Z", "endTime": "2026-08-25T11:39:44.596654856Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-nkwvwj" for cluster "cluster-candidate-5-0-nkwvwj" Unexpected error: <fmt.wrapError \| 0xc000179060>: failed to create NodePool np-nkwvwj: failed waiting for nodepool="np-nkwvwj" for cluster "cluster-candidate-5-0-nkwvwj" in resourcegroup="rg-candidate-5-0-qh5dlq8h25tn" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/02a0680c-d02a-4979-935f-a2788b9917ce -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/02a0680c-d02a-4979-935f-a2788b9917ce", "name": "02a0680c-d02a-4979-935f-a2788b9917ce", "status": "Failed", "startTime": "2026-08-25T11:20:40.33393292Z", "endTime": "2026-08-25T11:39:44.596654856Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [ingress-operator]; provider Microsoft.RedHatOpenShift 10:22:57 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc000af62e0>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-5j68lm294d7v" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e4433c13-3a16-45ba-88e7-ec043217acd4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e4433c13-3a16-45ba-88e7-ec043217acd4", "name": "e4433c13-3a16-45ba-88e7-ec043217acd4", "status": "Failed", "startTime": "2026-08-25T11:01:40.830511566Z", "endTime": "2026-08-25T11:20:48.170023371Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator; hosted cluster degraded: UnavailableReplicas: ingress-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc000af62e0>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-5j68lm294d7v" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e4433c13-3a16-45ba-88e7-ec043217acd4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e4433c13-3a16-45ba-88e7-ec043217acd4", "name": "e4433c13-3a16-45ba-88e7-ec043217acd4", "status": "Failed", "startTime": "2026-08-25T11:01:40.830511566Z", "endTime": "2026-08-25T11:20:48.170023371Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator; hosted cluster degraded: UnavailableReplicas: ingress-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 10:22:57 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints Unexpected error: <fmt.wrapError \| 0xc0001b7840>: failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-tz75dw8glcvr" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/83546c4c-4fc8-4a2c-89ac-1ab5e50bf205 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/83546c4c-4fc8-4a2c-89ac-1ab5e50bf205", "name": "83546c4c-4fc8-4a2c-89ac-1ab5e50bf205", "status": "Failed", "startTime": "2026-08-25T11:19:39.701946691Z", "endTime": "2026-08-25T11:38:44.54436442Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints Unexpected error: <fmt.wrapError \| 0xc0001b7840>: failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-tz75dw8glcvr" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/83546c4c-4fc8-4a2c-89ac-1ab5e50bf205 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/83546c4c-4fc8-4a2c-89ac-1ab5e50bf205", "name": "83546c4c-4fc8-4a2c-89ac-1ab5e50bf205", "status": "Failed", "startTime": "2026-08-25T11:19:39.701946691Z", "endTime": "2026-08-25T11:38:44.54436442Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [ingress-operator]; provider Microsoft.RedHatOpenShift 10:22:57 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-bp8trx" with version 5.0.0-ec.5 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000d85000>: failed to create HCP cluster cluster-zstream-5-0-bp8trx: failed waiting for cluster="cluster-zstream-5-0-bp8trx" in resourcegroup="rg-zstream-upgrade-5-0-brkg7nwpr7jw" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3b253226-385a-4503-a738-5e57c00054ba -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3b253226-385a-4503-a738-5e57c00054ba", "name": "3b253226-385a-4503-a738-5e57c00054ba", "status": "Failed", "startTime": "2026-08-25T11:01:19.290629965Z", "endTime": "2026-08-25T11:20:27.735529124Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator; hosted cluster degraded: UnavailableReplicas: ingress-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-bp8trx" with version 5.0.0-ec.5 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000d85000>: failed to create HCP cluster cluster-zstream-5-0-bp8trx: failed waiting for cluster="cluster-zstream-5-0-bp8trx" in resourcegroup="rg-zstream-upgrade-5-0-brkg7nwpr7jw" to finish creating: GET https://rp.j9005312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3b253226-385a-4503-a738-5e57c00054ba -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3b253226-385a-4503-a738-5e57c00054ba", "name": "3b253226-385a-4503-a738-5e57c00054ba", "status": "Failed", "startTime": "2026-08-25T11:01:19.290629965Z", "endTime": "2026-08-25T11:20:27.735529124Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator; hosted cluster degraded: UnavailableReplicas: ingress-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (2) alert [svc] UJClusterProvisionErrors6h30m fired 10:22:57 UTC · failed at alert · test [aro-hcp-observability] [svc] alert UJClusterProvisionErrors6h30m does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-08-25T12:06:25Z Severity: Sev3 Labels: alertname="UJClusterProvisionErrors6h30m", cluster="ci01-j9005312-svc", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9005312/providers/microsoft.monitor/accounts/services-j9005312", severity="3", short_window="30m", slo="cluster-provision-errors" Description: More than 30% of cluster create (install) operations are in failed state sustained over 30 minutes, indicating a medium error budget burn (6x) that would exhaust the 95% SLO budget in ~28 hours. alert [hcp] KubePodNotReady fired 10:22:57 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 69 time(s) (22 unknown, 47 known) Firing 1: State: Resolved Started: 2026-08-25T11:12:48Z Ended: 2026-08-25T11:18:54Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9005312-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig2e8otp76oitb1jh1f4qm5u2spvk-w1t1q3s6x1e1x5d", pod="router-577849c6f-7p6c6", severity="warning" Description: Pod ocm-arohcpci01-2sdig2e8otp76oitb1jh1f4qm5u2spvk-w1t1q3s6x1e1x5d/router-577849c6f-7p6c6 has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-08-25T11:13:50Z Ended: 2026-08-25T11:20:53Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9005312-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig05qrvs2evs8a30l07oj4ofkolfp-l5a2z6l6l6v2z2w", pod="router-766f6797fd-4t9pk", severity="warning" Description: Pod ocm-arohcpci01-2sdig05qrvs2evs8a30l07oj4ofkolfp-l5a2z6l6l6v2z2w/router-766f6797fd-4t9pk has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-08-25T11:14:50Z Ended: 2026-08-25T11:22:51Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9005312-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig2sgfv21infum186h01mgb4tifri-j2m8e0t6e9r5w4s", pod="ingress-operator-c798f4587-xnfgv", severity="warning" Description: Pod ocm-arohcpci01-2sdig2sgfv21infum186h01mgb4tifri-j2m8e0t6e9r5w4s/ingress-operator-c798f4587-xnfgv has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-08-25T11:14:50Z Ended: 2026-08-25T11:23:50Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9005312-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdiftlm8brlvli4n9drmv1vhs3oeth6-z8n3t9a8v2e7e7m", pod="cluster-image-registry-operator-6f4bd7f774-wklr7", severity="warning" Description: Pod ocm-arohcpci01-2sdiftlm8brlvli4n9drmv1vhs3oeth6-z8n3t9a8v2e7e7m/cluster-image-registry-operator-6f4bd7f774-wklr7 has been in a non-ready state for longer than 5 minutes. Firing 5: State: Resolved Started: 2026-08-25T11:14:50Z Ended: 2026-08-25T11:21:48Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9005312-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig2v4aa1s2gfru3f1uv3jlivc0jtd-p5t5x8w8z0r0g4h", pod="etcd-2", severity="warning" Description: Pod ocm-arohcpci01-2sdig2v4aa1s2gfru3f1uv3jlivc0jtd-p5t5x8w8z0r0g4h/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 6: State: Resolved Started: 2026-08-25T11:14:50Z Ended: 2026-08-25T11:21:48Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9005312-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig2sgfv21infum186h01mgb4tifri-j2m8e0t6e9r5w4s", pod="cluster-network-operator-85f5d57c7d-82nbp", severity="warning" Description: Pod ocm-arohcpci01-2sdig2sgfv21infum186h01mgb4tifri-j2m8e0t6e9r5w4s/cluster-network-operator-85f5d57c7d-82nbp has been in a non-ready state for longer than 5 minutes. Firing 7: State: Resolved Started: 2026-08-25T11:14:50Z Ended: 2026-08-25T11:20:53Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9005312-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig2sgfv21infum186h01mgb4tifri-j2m8e0t6e9r5w4s", pod="cluster-image-registry-operator-986f78f74-7d477", severity="warning" Description: Pod ocm-arohcpci01-2sdig2sgfv21infum186h01mgb4tifri-j2m8e0t6e9r5w4s/cluster-image-registry-operator-986f78f74-7d477 has been in a non-ready state for longer than 5 minutes. Firing 8: State: Resolved Started: 2026-08-25T11:14:50Z Ended: 2026-08-25T11:21:48Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9005312-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdifubr6du1el2mlsfa6mhn6l93m64s-r9r3n5t4g5z1o6l", pod="cluster-network-operator-66c6fd486f-97mb6", severity="warning" Description: Pod ocm-arohcpci01-2sdifubr6du1el2mlsfa6mhn6l93m64s-r9r3n5t4g5z1o6l/cluster-network-operator-66c6fd486f-97mb6 has been in a non-ready state for longer than 5 minutes. Firing 9: State: Resolved Started: 2026-08-25T11:14:50Z Ended: 2026-08-25T11:22:51Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9005312-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig309ecpcs32nrdd7hs9n6tvobrvk-p6x2m3o7t1x5c7m", pod="ingress-operator-785d88785-xhpgl", severity="warning" Description: Pod ocm-arohcpci01-2sdig309ecpcs32nrdd7hs9n6tvobrvk-p6x2m3o7t1x5c7m/ingress-operator-785d88785-xhpgl has been in a non-ready state for longer than 5 minutes. Firing 10: State: Resolved Started: 2026-08-25T11:16:50Z Ended: 2026-08-25T11:23:50Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9005312-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdifiu3kfspedp7bg2adod09p8tjs51-m4h6a7s1g6y0y7w", pod="cluster-network-operator-867cf476f7-jhtbt", severity="warning" Description: Pod ocm-arohcpci01-2sdifiu3kfspedp7bg2adod09p8tjs51-m4h6a7s1g6y0y7w/cluster-network-operator-867cf476f7-jhtbt has been in a non-ready state for longer than 5 minutes. Firing 11: State: Resolved Started: 2026-08-25T11:16:50Z Ended: 2026-08-25T11:29:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9005312-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig02iob6e2k6b50c08hc14r75pioo-ea-listwl9qrq", pod="ingress-operator-6d5c4d9bf5-xzc5j", severity="warning" Description: Pod ocm-arohcpci01-2sdig02iob6e2k6b50c08hc14r75pioo-ea-listwl9qrq/ingress-operator-6d5c4d9bf5-xzc5j has been in a non-ready state for longer than 5 minutes. Firing 12: State: Resolved Started: 2026-08-25T11:16:50Z Ended: 2026-08-25T11:23:50Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9005312-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig02iob6e2k6b50c08hc14r75pioo-ea-listwl9qrq", pod="cluster-network-operator-854dbb4986-stn4n", severity="warning" Description: Pod ocm-arohcpci01-2sdig02iob6e2k6b50c08hc14r75pioo-ea-listwl9qrq/cluster-network-operator-854dbb4986-stn4n has been in a non-ready state for longer than 5 minutes. Firing 13: State: Resolved Started: 2026-08-25T11:16:50Z Ended: 2026-08-25T11:28:52Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9005312-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig527d27t8t672b9d13c306pbqg6a-y0p1b3l1a2l2q3k", pod="router-5658fb655-mr26f", severity="warning" Description: Pod ocm-arohcpci01-2sdig527d27t8t672b9d13c306pbqg6a-y0p1b3l1a2l2q3k/router-5658fb655-mr26f has been in a non-ready state for longer than 5 minutes. Firing 14: State: Resolved Started: 2026-08-25T11:16:50Z Ended: 2026-08-25T11:22:51Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9005312-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig02iob6e2k6b50c08hc14r75pioo-ea-listwl9qrq", pod="dns-operator-77874984f6-jqx8z", severity="warning" Description: Pod ocm-arohcpci01-2sdig02iob6e2k6b50c08hc14r75pioo-ea-listwl9qrq/dns-operator-77874984f6-jqx8z has been in a non-ready state for longer than 5 minutes. Firing 15: State: Resolved Started: 2026-08-25T11:16:50Z Ended: 2026-08-25T11:22:52Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9005312-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdifiu3kfspedp7bg2adod09p8tjs51-m4h6a7s1g6y0y7w", pod="ingress-operator-c7c7dc876-f9qx2", severity="warning" Description: Pod ocm-arohcpci01-2sdifiu3kfspedp7bg2adod09p8tjs51-m4h6a7s1g6y0y7w/ingress-operator-c7c7dc876-f9qx2 has been in a non-ready state for longer than 5 minutes. Firing 16: State: Resolved Started: 2026-08-25T11:16:50Z Ended: 2026-08-25T11:25:53Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9005312-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig23l8p6umhg7m2pe7183ddad2jbq-p6f4c6v2i1p4t1k", pod="router-69ff88c7b-xfcbr", severity="warning" Description: Pod ocm-arohcpci01-2sdig23l8p6umhg7m2pe7183ddad2jbq-p6f4c6v2i1p4t1k/router-69ff88c7b-xfcbr has been in a non-ready state for longer than 5 minutes. Firing 17: State: Resolved Started: 2026-08-25T11:17:49Z Ended: 2026-08-25T11:24:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9005312-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig2e8otp76oitb1jh1f4qm5u2spvk-w1t1q3s6x1e1x5d", pod="openshift-apiserver-5876f5948f-2bqfp", severity="warning" Description: Pod ocm-arohcpci01-2sdig2e8otp76oitb1jh1f4qm5u2spvk-w1t1q3s6x1e1x5d/openshift-apiserver-5876f5948f-2bqfp has been in a non-ready state for longer than 5 minutes. Firing 18: State: Resolved Started: 2026-08-25T11:17:49Z Ended: 2026-08-25T11:25:53Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9005312-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig2hpum7e5n4s0jib2t987cajn7pc-v8b8k5t9b3l5k2k", pod="ingress-operator-749cc6fb84-xrq6p", severity="warning" Description: Pod ocm-arohcpci01-2sdig2hpum7e5n4s0jib2t987cajn7pc-v8b8k5t9b3l5k2k/ingress-operator-749cc6fb84-xrq6p has been in a non-ready state for longer than 5 minutes. Firing 19: State: Resolved Started: 2026-08-25T11:17:50Z Ended: 2026-08-25T11:24:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9005312-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig2c4rftbtjejq4p3jigrj9s8vack-q2r5c0q7z3z1p6r", pod="etcd-0", severity="warning" Description: Pod ocm-arohcpci01-2sdig2c4rftbtjejq4p3jigrj9s8vack-q2r5c0q7z3z1p6r/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 20: State: Resolved Started: 2026-08-25T11:17:50Z Ended: 2026-08-25T11:24:48Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9005312-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdiftnonf8jdmjlr4jidiklg8rort12-e8t3p4z5l9d6i7m", pod="multus-admission-controller-696ff77bdb-mfcs6", severity="warning" Description: Pod ocm-arohcpci01-2sdiftnonf8jdmjlr4jidiklg8rort12-e8t3p4z5l9d6i7m/multus-admission-controller-696ff77bdb-mfcs6 has been in a non-ready state for longer than 5 minutes. Firing 21: State: Resolved Started: 2026-08-25T11:18:53Z Ended: 2026-08-25T11:24:48Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9005312-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig23l8p6umhg7m2pe7183ddad2jbq-p6f4c6v2i1p4t1k", pod="kube-apiserver-d85d65586-x7jxd", severity="warning" Description: Pod ocm-arohcpci01-2sdig23l8p6umhg7m2pe7183ddad2jbq-p6f4c6v2i1p4t1k/kube-apiserver-d85d65586-x7jxd has been in a non-ready state for longer than 5 minutes. Firing 22: State: Resolved Started: 2026-08-25T11:18:53Z Ended: 2026-08-25T11:30:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9005312-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig2e8otp76oitb1jh1f4qm5u2spvk-w1t1q3s6x1e1x5d", pod="router-7999d5c7bc-g6gx8", severity="warning" Description: Pod ocm-arohcpci01-2sdig2e8otp76oitb1jh1f4qm5u2spvk-w1t1q3s6x1e1x5d/router-7999d5c7bc-g6gx8 has been in a non-ready state for longer than 5 minutes.
10:18:15	pull-ci-Azure-ARO-HCP-main-e2e-parallel head a095cb3	alert, e2e	failed	⚠#6672 (open)	30	e2e (27) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:18:15 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling Unexpected error: <fmt.wrapError \| 0xc001018140>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-gvdtksqqw7j8" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f3e1242e-a94b-425d-8b4d-fa06d2347c55 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f3e1242e-a94b-425d-8b4d-fa06d2347c55", "name": "f3e1242e-a94b-425d-8b4d-fa06d2347c55", "status": "Failed", "startTime": "2026-08-25T11:01:42.464677853Z", "endTime": "2026-08-25T11:20:47.7869157Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling Unexpected error: <fmt.wrapError \| 0xc001018140>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-gvdtksqqw7j8" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f3e1242e-a94b-425d-8b4d-fa06d2347c55 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f3e1242e-a94b-425d-8b4d-fa06d2347c55", "name": "f3e1242e-a94b-425d-8b4d-fa06d2347c55", "status": "Failed", "startTime": "2026-08-25T11:01:42.464677853Z", "endTime": "2026-08-25T11:20:47.7869157Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 10:18:15 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc001178620>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-pp7257vrdlf5" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/64c0748d-e4ba-4941-92be-50ba2fb09d8b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/64c0748d-e4ba-4941-92be-50ba2fb09d8b", "name": "64c0748d-e4ba-4941-92be-50ba2fb09d8b", "status": "Failed", "startTime": "2026-08-25T11:02:02.431917892Z", "endTime": "2026-08-25T11:21:07.397091958Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc001178620>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-pp7257vrdlf5" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/64c0748d-e4ba-4941-92be-50ba2fb09d8b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/64c0748d-e4ba-4941-92be-50ba2fb09d8b", "name": "64c0748d-e4ba-4941-92be-50ba2fb09d8b", "status": "Failed", "startTime": "2026-08-25T11:02:02.431917892Z", "endTime": "2026-08-25T11:21:07.397091958Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:18:15 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0xc000900480>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-dgghghmcnzdm" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aa686f08-f4cc-4a79-8daa-f55b889c2e5b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aa686f08-f4cc-4a79-8daa-f55b889c2e5b", "name": "aa686f08-f4cc-4a79-8daa-f55b889c2e5b", "status": "Failed", "startTime": "2026-08-25T11:01:53.300970261Z", "endTime": "2026-08-25T11:20:57.373035491Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0xc000900480>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-dgghghmcnzdm" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aa686f08-f4cc-4a79-8daa-f55b889c2e5b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aa686f08-f4cc-4a79-8daa-f55b889c2e5b", "name": "aa686f08-f4cc-4a79-8daa-f55b889c2e5b", "status": "Failed", "startTime": "2026-08-25T11:01:53.300970261Z", "endTime": "2026-08-25T11:20:57.373035491Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-controller-manager]; provider Microsoft.RedHatOpenShift 10:18:15 UTC · failed at e2e · test SRE can pause schedules to stop backup execution for an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000855700>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-8spmvpshpxg4" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8e884ed9-970f-40fe-b266-642f03ffe736 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8e884ed9-970f-40fe-b266-642f03ffe736", "name": "8e884ed9-970f-40fe-b266-642f03ffe736", "status": "Failed", "startTime": "2026-08-25T11:01:52.700430635Z", "endTime": "2026-08-25T11:20:57.397456607Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 2 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000855700>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-8spmvpshpxg4" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8e884ed9-970f-40fe-b266-642f03ffe736 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8e884ed9-970f-40fe-b266-642f03ffe736", "name": "8e884ed9-970f-40fe-b266-642f03ffe736", "status": "Failed", "startTime": "2026-08-25T11:01:52.700430635Z", "endTime": "2026-08-25T11:20:57.397456607Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 2 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator, ignition-server-proxy]; provider Microsoft.RedHatOpenShift 10:18:15 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260630preview and verify the ingress is internal · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc000d02900>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-sn9l5gh5sgnq" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/77d8a5d2-c15f-4303-9203-ed2a4dc77c20 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/77d8a5d2-c15f-4303-9203-ed2a4dc77c20", "name": "77d8a5d2-c15f-4303-9203-ed2a4dc77c20", "status": "Failed", "startTime": "2026-08-25T11:02:02.408692636Z", "endTime": "2026-08-25T11:21:07.333560396Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc000d02900>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-sn9l5gh5sgnq" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/77d8a5d2-c15f-4303-9203-ed2a4dc77c20 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/77d8a5d2-c15f-4303-9203-ed2a4dc77c20", "name": "77d8a5d2-c15f-4303-9203-ed2a4dc77c20", "status": "Failed", "startTime": "2026-08-25T11:02:02.408692636Z", "endTime": "2026-08-25T11:21:07.333560396Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 10:18:15 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-fhgqh26vqnp9/cluster-candidate-4-22-lw4fjm should provision Unexpected error: <fmt.wrapError \| 0xc001a02400>: failed to create HCP cluster cluster-candidate-4-22-lw4fjm: failed waiting for cluster="cluster-candidate-4-22-lw4fjm" in resourcegroup="rg-candidate-4-22-fhgqh26vqnp9" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/99c474d9-a201-4fec-b07b-8d9819fe8425 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/99c474d9-a201-4fec-b07b-8d9819fe8425", "name": "99c474d9-a201-4fec-b07b-8d9819fe8425", "status": "Failed", "startTime": "2026-08-25T11:01:52.421892504Z", "endTime": "2026-08-25T11:20:57.269333787Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, cluster-storage-operator, catalog-operator, certified-operators-catalog, olm-operator, packageserver, hosted-cluster-config-operator, ingress-operator, community-operators-catalog, csi-snapshot-controller-operator, redhat-operators-catalog, cluster-network-operator, cluster-image-registry-operator, dns-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-fhgqh26vqnp9/cluster-candidate-4-22-lw4fjm should provision Unexpected error: <fmt.wrapError \| 0xc001a02400>: failed to create HCP cluster cluster-candidate-4-22-lw4fjm: failed waiting for cluster="cluster-candidate-4-22-lw4fjm" in resourcegroup="rg-candidate-4-22-fhgqh26vqnp9" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/99c474d9-a201-4fec-b07b-8d9819fe8425 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/99c474d9-a201-4fec-b07b-8d9819fe8425", "name": "99c474d9-a201-4fec-b07b-8d9819fe8425", "status": "Failed", "startTime": "2026-08-25T11:01:52.421892504Z", "endTime": "2026-08-25T11:20:57.269333787Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, cluster-storage-operator, catalog-operator, certified-operators-catalog, olm-operator, packageserver, hosted-cluster-config-operator, ingress-operator, community-operators-catalog, csi-snapshot-controller-operator, redhat-operators-catalog, cluster-network-operator, cluster-image-registry-operator, dns-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator, ingress-operator, packageserver]; provider Microsoft.RedHatOpenShift 10:18:15 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:721]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc0014868a0>: failed to create HCP cluster np-dg-minor-6lfgc2: failed waiting for cluster="np-dg-minor-6lfgc2" in resourcegroup="rg-np-dg-minor-6lfgc2-lwtz9m4rnlh9" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/169a9726-cf09-492c-bf9a-53259660198c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/169a9726-cf09-492c-bf9a-53259660198c", "name": "169a9726-cf09-492c-bf9a-53259660198c", "status": "Failed", "startTime": "2026-08-25T11:01:53.246951809Z", "endTime": "2026-08-25T11:20:57.403791086Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:721]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc0014868a0>: failed to create HCP cluster np-dg-minor-6lfgc2: failed waiting for cluster="np-dg-minor-6lfgc2" in resourcegroup="rg-np-dg-minor-6lfgc2-lwtz9m4rnlh9" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/169a9726-cf09-492c-bf9a-53259660198c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/169a9726-cf09-492c-bf9a-53259660198c", "name": "169a9726-cf09-492c-bf9a-53259660198c", "status": "Failed", "startTime": "2026-08-25T11:01:53.246951809Z", "endTime": "2026-08-25T11:20:57.403791086Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift 10:18:15 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc0014d4ba0>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-djpx8vzhvdqj" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a29c2b48-aa8b-4185-a975-2ba36d80abfa -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a29c2b48-aa8b-4185-a975-2ba36d80abfa", "name": "a29c2b48-aa8b-4185-a975-2ba36d80abfa", "status": "Failed", "startTime": "2026-08-25T11:01:32.971395174Z", "endTime": "2026-08-25T11:20:37.411972437Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, control-plane-pki-operator, cluster-storage-operator, dns-operator, ingress-operator, ignition-server, olm-collect-profiles, certified-operators-catalog, kube-controller-manager, openshift-apiserver, cluster-version-operator, openshift-controller-manager, openshift-route-controller-manager, cluster-network-operator, konnectivity-agent, ignition-server-proxy, olm-operator, redhat-operators-catalog, redhat-marketplace-catalog, community-operators-catalog, cluster-autoscaler, machine-approver, packageserver, kube-scheduler, azure-cloud-controller-manager, cluster-image-registry-operator, catalog-operator, cluster-policy-controller, hosted-cluster-config-operator, cluster-node-tuning-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc0014d4ba0>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-djpx8vzhvdqj" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a29c2b48-aa8b-4185-a975-2ba36d80abfa -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a29c2b48-aa8b-4185-a975-2ba36d80abfa", "name": "a29c2b48-aa8b-4185-a975-2ba36d80abfa", "status": "Failed", "startTime": "2026-08-25T11:01:32.971395174Z", "endTime": "2026-08-25T11:20:37.411972437Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, control-plane-pki-operator, cluster-storage-operator, dns-operator, ingress-operator, ignition-server, olm-collect-profiles, certified-operators-catalog, kube-controller-manager, openshift-apiserver, cluster-version-operator, openshift-controller-manager, openshift-route-controller-manager, cluster-network-operator, konnectivity-agent, ignition-server-proxy, olm-operator, redhat-operators-catalog, redhat-marketplace-catalog, community-operators-catalog, cluster-autoscaler, machine-approver, packageserver, kube-scheduler, azure-cloud-controller-manager, cluster-image-registry-operator, catalog-operator, cluster-policy-controller, hosted-cluster-config-operator, cluster-node-tuning-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:18:15 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-mbbxgn with version 4.21.30 Unexpected error: <fmt.wrapError \| 0xc000fbe500>: failed to create HCP cluster np-version-upgrade-cluster-mbbxgn: failed waiting for cluster="np-version-upgrade-cluster-mbbxgn" in resourcegroup="rg-np-version-upgrade-mbbxgn-djj4c2xjbmdj" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/60437c45-38ab-47e8-b7f9-9c016655bbda -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/60437c45-38ab-47e8-b7f9-9c016655bbda", "name": "60437c45-38ab-47e8-b7f9-9c016655bbda", "status": "Failed", "startTime": "2026-08-25T11:01:42.841852986Z", "endTime": "2026-08-25T11:20:47.887806017Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-mbbxgn with version 4.21.30 Unexpected error: <fmt.wrapError \| 0xc000fbe500>: failed to create HCP cluster np-version-upgrade-cluster-mbbxgn: failed waiting for cluster="np-version-upgrade-cluster-mbbxgn" in resourcegroup="rg-np-version-upgrade-mbbxgn-djj4c2xjbmdj" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/60437c45-38ab-47e8-b7f9-9c016655bbda -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/60437c45-38ab-47e8-b7f9-9c016655bbda", "name": "60437c45-38ab-47e8-b7f9-9c016655bbda", "status": "Failed", "startTime": "2026-08-25T11:01:42.841852986Z", "endTime": "2026-08-25T11:20:47.887806017Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator]; provider Microsoft.RedHatOpenShift 10:18:15 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc000c70e60>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-j2h7qqprw2cq" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d441ebfa-adad-47e3-810c-f2ff28ffd4b3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d441ebfa-adad-47e3-810c-f2ff28ffd4b3", "name": "d441ebfa-adad-47e3-810c-f2ff28ffd4b3", "status": "Failed", "startTime": "2026-08-25T11:04:15.897556457Z", "endTime": "2026-08-25T11:23:17.388317431Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator; hosted cluster degraded: UnavailableReplicas: cluster-network-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc000c70e60>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-j2h7qqprw2cq" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d441ebfa-adad-47e3-810c-f2ff28ffd4b3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d441ebfa-adad-47e3-810c-f2ff28ffd4b3", "name": "d441ebfa-adad-47e3-810c-f2ff28ffd4b3", "status": "Failed", "startTime": "2026-08-25T11:04:15.897556457Z", "endTime": "2026-08-25T11:23:17.388317431Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator; hosted cluster degraded: UnavailableReplicas: cluster-network-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, ingress-operator]; provider Microsoft.RedHatOpenShift 10:18:15 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:98]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0xc0013da220>: failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-hk9dqp59j9m5" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cb6081e4-ae2a-4eab-a6f8-42f62611bc3c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cb6081e4-ae2a-4eab-a6f8-42f62611bc3c", "name": "cb6081e4-ae2a-4eab-a6f8-42f62611bc3c", "status": "Failed", "startTime": "2026-08-25T11:01:54.161881545Z", "endTime": "2026-08-25T11:20:57.408325813Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, ingress-operator; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:98]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0xc0013da220>: failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-hk9dqp59j9m5" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cb6081e4-ae2a-4eab-a6f8-42f62611bc3c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cb6081e4-ae2a-4eab-a6f8-42f62611bc3c", "name": "cb6081e4-ae2a-4eab-a6f8-42f62611bc3c", "status": "Failed", "startTime": "2026-08-25T11:01:54.161881545Z", "endTime": "2026-08-25T11:20:57.408325813Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, ingress-operator; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [ingress-operator, router]; provider Microsoft.RedHatOpenShift 10:18:15 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-rc5jrpz77lgk/cluster-candidate-5-0-k8xlxd should provision Unexpected error: <fmt.wrapError \| 0xc001060a40>: failed to create HCP cluster cluster-candidate-5-0-k8xlxd: failed waiting for cluster="cluster-candidate-5-0-k8xlxd" in resourcegroup="rg-candidate-5-0-rc5jrpz77lgk" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/223cfa5a-75b0-4cb9-b839-dc5100d41095 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/223cfa5a-75b0-4cb9-b839-dc5100d41095", "name": "223cfa5a-75b0-4cb9-b839-dc5100d41095", "status": "Failed", "startTime": "2026-08-25T11:01:32.648244013Z", "endTime": "2026-08-25T11:20:37.341589368Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator; hosted cluster degraded: UnavailableReplicas: [ingress-operator deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-rc5jrpz77lgk/cluster-candidate-5-0-k8xlxd should provision Unexpected error: <fmt.wrapError \| 0xc001060a40>: failed to create HCP cluster cluster-candidate-5-0-k8xlxd: failed waiting for cluster="cluster-candidate-5-0-k8xlxd" in resourcegroup="rg-candidate-5-0-rc5jrpz77lgk" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/223cfa5a-75b0-4cb9-b839-dc5100d41095 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/223cfa5a-75b0-4cb9-b839-dc5100d41095", "name": "223cfa5a-75b0-4cb9-b839-dc5100d41095", "status": "Failed", "startTime": "2026-08-25T11:01:32.648244013Z", "endTime": "2026-08-25T11:20:37.341589368Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator; hosted cluster degraded: UnavailableReplicas: [ingress-operator deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [ingress-operator, openshift-route-controller-manager]; provider Microsoft.RedHatOpenShift 10:18:15 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc000706220>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-tdgbq5vc5q77" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e298fc5b-ea25-4c91-afe3-b4998ff1e5f2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e298fc5b-ea25-4c91-afe3-b4998ff1e5f2", "name": "e298fc5b-ea25-4c91-afe3-b4998ff1e5f2", "status": "Failed", "startTime": "2026-08-25T11:02:45.146044639Z", "endTime": "2026-08-25T11:21:47.576163129Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, ingress-operator; hosted cluster degraded: UnavailableReplicas: [ingress-operator deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 2 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc000706220>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-tdgbq5vc5q77" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e298fc5b-ea25-4c91-afe3-b4998ff1e5f2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e298fc5b-ea25-4c91-afe3-b4998ff1e5f2", "name": "e298fc5b-ea25-4c91-afe3-b4998ff1e5f2", "status": "Failed", "startTime": "2026-08-25T11:02:45.146044639Z", "endTime": "2026-08-25T11:21:47.576163129Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, ingress-operator; hosted cluster degraded: UnavailableReplicas: [ingress-operator deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 2 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift 10:18:15 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: failed to create HCP cluster np-update-nodes-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc0012a2260>: failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-kp69jsvl6hm4" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b6e6c7c1-86af-4757-abbe-ed454d6ac3b8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b6e6c7c1-86af-4757-abbe-ed454d6ac3b8", "name": "b6e6c7c1-86af-4757-abbe-ed454d6ac3b8", "status": "Failed", "startTime": "2026-08-25T11:01:44.186307399Z", "endTime": "2026-08-25T11:20:47.798437893Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: failed to create HCP cluster np-update-nodes-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc0012a2260>: failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-kp69jsvl6hm4" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b6e6c7c1-86af-4757-abbe-ed454d6ac3b8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b6e6c7c1-86af-4757-abbe-ed454d6ac3b8", "name": "b6e6c7c1-86af-4757-abbe-ed454d6ac3b8", "status": "Failed", "startTime": "2026-08-25T11:01:44.186307399Z", "endTime": "2026-08-25T11:20:47.798437893Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 10:18:15 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0xc00062e1c0>: failed to create HCP cluster ea-listqqp748: failed waiting for cluster="ea-listqqp748" in resourcegroup="ea-list-t4fnk5lsvq88" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c1a2b240-6346-4e98-b4bb-870288292b46 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c1a2b240-6346-4e98-b4bb-870288292b46", "name": "c1a2b240-6346-4e98-b4bb-870288292b46", "status": "Failed", "startTime": "2026-08-25T11:02:13.223545992Z", "endTime": "2026-08-25T11:21:17.3486915Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, cluster-network-operator, csi-snapshot-controller-operator, openshift-controller-manager, olm-operator, cluster-image-registry-operator, catalog-operator, openshift-route-controller-manager, cluster-policy-controller, cluster-version-operator, packageserver, hosted-cluster-config-operator, cluster-node-tuning-operator, dns-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 2 unavailable replicas, openshift-route-controller-manager deployment has 2 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0xc00062e1c0>: failed to create HCP cluster ea-listqqp748: failed waiting for cluster="ea-listqqp748" in resourcegroup="ea-list-t4fnk5lsvq88" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c1a2b240-6346-4e98-b4bb-870288292b46 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c1a2b240-6346-4e98-b4bb-870288292b46", "name": "c1a2b240-6346-4e98-b4bb-870288292b46", "status": "Failed", "startTime": "2026-08-25T11:02:13.223545992Z", "endTime": "2026-08-25T11:21:17.3486915Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, cluster-network-operator, csi-snapshot-controller-operator, openshift-controller-manager, olm-operator, cluster-image-registry-operator, catalog-operator, openshift-route-controller-manager, cluster-policy-controller, cluster-version-operator, packageserver, hosted-cluster-config-operator, cluster-node-tuning-operator, dns-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 2 unavailable replicas, openshift-route-controller-manager deployment has 2 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, cluster-network-operator, csi-snapshot-controller-operator, dns-operator, hosted-cluster-config-operator, ingress-operator, olm-operator, packageserver]; provider Microsoft.RedHatOpenShift 10:18:15 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc000ea1740>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-jm4vsb7vn4sd" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/afbb1cd8-1c39-42be-9ff7-787ffc89e069 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/afbb1cd8-1c39-42be-9ff7-787ffc89e069", "name": "afbb1cd8-1c39-42be-9ff7-787ffc89e069", "status": "Failed", "startTime": "2026-08-25T11:03:22.824480944Z", "endTime": "2026-08-25T11:22:27.407570836Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, csi-snapshot-controller-operator, cluster-version-operator, olm-operator, dns-operator, catalog-operator, cluster-storage-operator, ingress-operator, hosted-cluster-config-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc000ea1740>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-jm4vsb7vn4sd" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/afbb1cd8-1c39-42be-9ff7-787ffc89e069 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/afbb1cd8-1c39-42be-9ff7-787ffc89e069", "name": "afbb1cd8-1c39-42be-9ff7-787ffc89e069", "status": "Failed", "startTime": "2026-08-25T11:03:22.824480944Z", "endTime": "2026-08-25T11:22:27.407570836Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, csi-snapshot-controller-operator, cluster-version-operator, olm-operator, dns-operator, catalog-operator, cluster-storage-operator, ingress-operator, hosted-cluster-config-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift 10:18:15 UTC · failed at e2e · test FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc00097b680>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-wgz7h9k2skjd" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a299a2cd-ef58-4c09-8fa3-3315683149e1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a299a2cd-ef58-4c09-8fa3-3315683149e1", "name": "a299a2cd-ef58-4c09-8fa3-3315683149e1", "status": "Failed", "startTime": "2026-08-25T11:01:52.19619894Z", "endTime": "2026-08-25T11:20:57.271055091Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc00097b680>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-wgz7h9k2skjd" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a299a2cd-ef58-4c09-8fa3-3315683149e1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a299a2cd-ef58-4c09-8fa3-3315683149e1", "name": "a299a2cd-ef58-4c09-8fa3-3315683149e1", "status": "Failed", "startTime": "2026-08-25T11:01:52.19619894Z", "endTime": "2026-08-25T11:20:57.271055091Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred Cluster provisioning failed 10:18:15 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager, catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver, router]; provider Microsoft.RedHatOpenShift 10:18:15 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--mdtqq6tfggzt/cluster-hs-pre-5md8sn should provision Unexpected error: <fmt.wrapError \| 0xc00138c380>: failed to create HCP cluster cluster-hs-pre-5md8sn: failed waiting for cluster="cluster-hs-pre-5md8sn" in resourcegroup="rg-hs-pre--mdtqq6tfggzt" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/faa7138f-d176-4225-9189-5771c1682daa -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/faa7138f-d176-4225-9189-5771c1682daa", "name": "faa7138f-d176-4225-9189-5771c1682daa", "status": "Failed", "startTime": "2026-08-25T11:01:52.650546675Z", "endTime": "2026-08-25T11:20:57.345950087Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator, ingress-operator, redhat-marketplace-catalog, cluster-policy-controller, azure-cloud-controller-manager, cluster-image-registry-operator, certified-operators-catalog, packageserver, cluster-version-operator, cluster-storage-operator, redhat-operators-catalog, csi-snapshot-controller-operator, openshift-controller-manager, cluster-node-tuning-operator, openshift-apiserver, catalog-operator, community-operators-catalog, openshift-route-controller-manager, hosted-cluster-config-operator, cluster-network-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 3 unavailable replicas, packageserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--mdtqq6tfggzt/cluster-hs-pre-5md8sn should provision Unexpected error: <fmt.wrapError \| 0xc00138c380>: failed to create HCP cluster cluster-hs-pre-5md8sn: failed waiting for cluster="cluster-hs-pre-5md8sn" in resourcegroup="rg-hs-pre--mdtqq6tfggzt" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/faa7138f-d176-4225-9189-5771c1682daa -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/faa7138f-d176-4225-9189-5771c1682daa", "name": "faa7138f-d176-4225-9189-5771c1682daa", "status": "Failed", "startTime": "2026-08-25T11:01:52.650546675Z", "endTime": "2026-08-25T11:20:57.345950087Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator, ingress-operator, redhat-marketplace-catalog, cluster-policy-controller, azure-cloud-controller-manager, cluster-image-registry-operator, certified-operators-catalog, packageserver, cluster-version-operator, cluster-storage-operator, redhat-operators-catalog, csi-snapshot-controller-operator, openshift-controller-manager, cluster-node-tuning-operator, openshift-apiserver, catalog-operator, community-operators-catalog, openshift-route-controller-manager, hosted-cluster-config-operator, cluster-network-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 3 unavailable replicas, packageserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred failed to read rendered config /opt/app-root/src/github.com/Azure/ARO-HCP/config/rendered/dev/ci01/canadacentral.yaml: open /opt/app-root/src/github.com/Azure/ARO-HCP/config/rendered/dev/ci01/canadacentral.yaml: no such file or directory 10:18:15 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:95]: failed to load rendered config — is RENDERED_CONFIG set? Unexpected error: <fmt.wrapError \| 0xc000eafac0>: failed to read rendered config /opt/app-root/src/github.com/Azure/ARO-HCP/config/rendered/dev/ci01/canadacentral.yaml: open /opt/app-root/src/github.com/Azure/ARO-HCP/config/rendered/dev/ci01/canadacentral.yaml: no such file or directory ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:95]: failed to load rendered config — is RENDERED_CONFIG set? Unexpected error: <fmt.wrapError \| 0xc000eafac0>: failed to read rendered config /opt/app-root/src/github.com/Azure/ARO-HCP/config/rendered/dev/ci01/canadacentral.yaml: open /opt/app-root/src/github.com/Azure/ARO-HCP/config/rendered/dev/ci01/canadacentral.yaml: no such file or directory ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator, ingress-operator]; provider Microsoft.RedHatOpenShift 10:18:15 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-5vxfzn with version 4.21.30 Unexpected error: <fmt.wrapError \| 0xc0005eed00>: failed to create HCP cluster np-version-upgrade-cluster-5vxfzn: failed waiting for cluster="np-version-upgrade-cluster-5vxfzn" in resourcegroup="rg-np-version-upgrade-5vxfzn-7svm57fp7wk7" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1fa03eeb-ac17-4494-af34-d2fa27ce5596 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1fa03eeb-ac17-4494-af34-d2fa27ce5596", "name": "1fa03eeb-ac17-4494-af34-d2fa27ce5596", "status": "Failed", "startTime": "2026-08-25T11:01:54.133785755Z", "endTime": "2026-08-25T11:20:57.201457228Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator, olm-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-5vxfzn with version 4.21.30 Unexpected error: <fmt.wrapError \| 0xc0005eed00>: failed to create HCP cluster np-version-upgrade-cluster-5vxfzn: failed waiting for cluster="np-version-upgrade-cluster-5vxfzn" in resourcegroup="rg-np-version-upgrade-5vxfzn-7svm57fp7wk7" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1fa03eeb-ac17-4494-af34-d2fa27ce5596 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1fa03eeb-ac17-4494-af34-d2fa27ce5596", "name": "1fa03eeb-ac17-4494-af34-d2fa27ce5596", "status": "Failed", "startTime": "2026-08-25T11:01:54.133785755Z", "endTime": "2026-08-25T11:20:57.201457228Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator, olm-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 10:18:15 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc000b14300>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-nkvt2c5pdsrv" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ecd5f2d-dd8e-444c-9b57-369de1b270b8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ecd5f2d-dd8e-444c-9b57-369de1b270b8", "name": "5ecd5f2d-dd8e-444c-9b57-369de1b270b8", "status": "Failed", "startTime": "2026-08-25T11:01:53.287776651Z", "endTime": "2026-08-25T11:20:57.385655782Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, olm-operator, hosted-cluster-config-operator, cluster-network-operator, packageserver, dns-operator, cluster-image-registry-operator, redhat-operators-catalog, catalog-operator, community-operators-catalog, ingress-operator, cluster-storage-operator, certified-operators-catalog, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc000b14300>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-nkvt2c5pdsrv" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ecd5f2d-dd8e-444c-9b57-369de1b270b8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ecd5f2d-dd8e-444c-9b57-369de1b270b8", "name": "5ecd5f2d-dd8e-444c-9b57-369de1b270b8", "status": "Failed", "startTime": "2026-08-25T11:01:53.287776651Z", "endTime": "2026-08-25T11:20:57.385655782Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, olm-operator, hosted-cluster-config-operator, cluster-network-operator, packageserver, dns-operator, cluster-image-registry-operator, redhat-operators-catalog, catalog-operator, community-operators-catalog, ingress-operator, cluster-storage-operator, certified-operators-catalog, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 10:18:15 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-psp25pblz7q9/cluster-candidate-4-20-5xxwvr should provision Unexpected error: <fmt.wrapError \| 0xc000947020>: failed to create HCP cluster cluster-candidate-4-20-5xxwvr: failed waiting for cluster="cluster-candidate-4-20-5xxwvr" in resourcegroup="rg-candidate-4-20-psp25pblz7q9" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ac623cd0-8df5-45d0-b2af-bc97016cb511 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ac623cd0-8df5-45d0-b2af-bc97016cb511", "name": "ac623cd0-8df5-45d0-b2af-bc97016cb511", "status": "Failed", "startTime": "2026-08-25T11:01:52.648184474Z", "endTime": "2026-08-25T11:20:57.398865961Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-psp25pblz7q9/cluster-candidate-4-20-5xxwvr should provision Unexpected error: <fmt.wrapError \| 0xc000947020>: failed to create HCP cluster cluster-candidate-4-20-5xxwvr: failed waiting for cluster="cluster-candidate-4-20-5xxwvr" in resourcegroup="rg-candidate-4-20-psp25pblz7q9" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ac623cd0-8df5-45d0-b2af-bc97016cb511 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ac623cd0-8df5-45d0-b2af-bc97016cb511", "name": "ac623cd0-8df5-45d0-b2af-bc97016cb511", "status": "Failed", "startTime": "2026-08-25T11:01:52.648184474Z", "endTime": "2026-08-25T11:20:57.398865961Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:18:15 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:147]: failed to create HCP cluster "cp-ystream-upgrade-4-22-qs7l5q" at install version 4.21 Unexpected error: <fmt.wrapError \| 0xc0016408c0>: failed to create HCP cluster cp-ystream-upgrade-4-22-qs7l5q: failed waiting for cluster="cp-ystream-upgrade-4-22-qs7l5q" in resourcegroup="rg-cp-ystream-upgrade-4-22-sn26t59hv7kg" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e7b354b0-edaa-4c54-8a45-d3577720bedd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e7b354b0-edaa-4c54-8a45-d3577720bedd", "name": "e7b354b0-edaa-4c54-8a45-d3577720bedd", "status": "Failed", "startTime": "2026-08-25T11:01:33.430099239Z", "endTime": "2026-08-25T11:20:37.33551722Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:147]: failed to create HCP cluster "cp-ystream-upgrade-4-22-qs7l5q" at install version 4.21 Unexpected error: <fmt.wrapError \| 0xc0016408c0>: failed to create HCP cluster cp-ystream-upgrade-4-22-qs7l5q: failed waiting for cluster="cp-ystream-upgrade-4-22-qs7l5q" in resourcegroup="rg-cp-ystream-upgrade-4-22-sn26t59hv7kg" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e7b354b0-edaa-4c54-8a45-d3577720bedd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e7b354b0-edaa-4c54-8a45-d3577720bedd", "name": "e7b354b0-edaa-4c54-8a45-d3577720bedd", "status": "Failed", "startTime": "2026-08-25T11:01:33.430099239Z", "endTime": "2026-08-25T11:20:37.33551722Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [csi-snapshot-controller-operator, dns-operator, ignition-server-proxy, packageserver, router]; provider Microsoft.RedHatOpenShift 10:18:15 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-n44w2df8vj8t/cluster-candidate-4-21-mppmvd should provision Unexpected error: <fmt.wrapError \| 0xc0010c4140>: failed to create HCP cluster cluster-candidate-4-21-mppmvd: failed waiting for cluster="cluster-candidate-4-21-mppmvd" in resourcegroup="rg-candidate-4-21-n44w2df8vj8t" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ef974da-bfca-4b86-a5a7-f5f72d3b3ecc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ef974da-bfca-4b86-a5a7-f5f72d3b3ecc", "name": "5ef974da-bfca-4b86-a5a7-f5f72d3b3ecc", "status": "Failed", "startTime": "2026-08-25T11:01:52.19951964Z", "endTime": "2026-08-25T11:20:57.290621961Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator, csi-snapshot-controller-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, packageserver deployment has 2 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-n44w2df8vj8t/cluster-candidate-4-21-mppmvd should provision Unexpected error: <fmt.wrapError \| 0xc0010c4140>: failed to create HCP cluster cluster-candidate-4-21-mppmvd: failed waiting for cluster="cluster-candidate-4-21-mppmvd" in resourcegroup="rg-candidate-4-21-n44w2df8vj8t" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ef974da-bfca-4b86-a5a7-f5f72d3b3ecc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ef974da-bfca-4b86-a5a7-f5f72d3b3ecc", "name": "5ef974da-bfca-4b86-a5a7-f5f72d3b3ecc", "status": "Failed", "startTime": "2026-08-25T11:01:52.19951964Z", "endTime": "2026-08-25T11:20:57.290621961Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator, csi-snapshot-controller-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, packageserver deployment has 2 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator, cluster-network-operator]; provider Microsoft.RedHatOpenShift 10:18:15 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000e0a6c0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-9mf2mfq4r2d9" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d8a52f10-9e35-4b81-8bed-515ddc39f2b4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d8a52f10-9e35-4b81-8bed-515ddc39f2b4", "name": "d8a52f10-9e35-4b81-8bed-515ddc39f2b4", "status": "Failed", "startTime": "2026-08-25T11:03:12.867884764Z", "endTime": "2026-08-25T11:22:17.332591987Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000e0a6c0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-9mf2mfq4r2d9" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d8a52f10-9e35-4b81-8bed-515ddc39f2b4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d8a52f10-9e35-4b81-8bed-515ddc39f2b4", "name": "d8a52f10-9e35-4b81-8bed-515ddc39f2b4", "status": "Failed", "startTime": "2026-08-25T11:03:12.867884764Z", "endTime": "2026-08-25T11:22:17.332591987Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:18:15 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-twq2f7" with version 4.20.34 on candidate channel Unexpected error: <fmt.wrapError \| 0xc0008b8240>: failed to create HCP cluster cluster-zstream-4-20-twq2f7: failed waiting for cluster="cluster-zstream-4-20-twq2f7" in resourcegroup="rg-zstream-upgrade-4-20-vnnwhcdv49hn" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b7a9b211-70f5-46b5-938d-82c5e0f7d27f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b7a9b211-70f5-46b5-938d-82c5e0f7d27f", "name": "b7a9b211-70f5-46b5-938d-82c5e0f7d27f", "status": "Failed", "startTime": "2026-08-25T11:01:52.235664358Z", "endTime": "2026-08-25T11:20:57.276791645Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-twq2f7" with version 4.20.34 on candidate channel Unexpected error: <fmt.wrapError \| 0xc0008b8240>: failed to create HCP cluster cluster-zstream-4-20-twq2f7: failed waiting for cluster="cluster-zstream-4-20-twq2f7" in resourcegroup="rg-zstream-upgrade-4-20-vnnwhcdv49hn" to finish creating: GET https://rp.j2151936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b7a9b211-70f5-46b5-938d-82c5e0f7d27f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b7a9b211-70f5-46b5-938d-82c5e0f7d27f", "name": "b7a9b211-70f5-46b5-938d-82c5e0f7d27f", "status": "Failed", "startTime": "2026-08-25T11:01:52.235664358Z", "endTime": "2026-08-25T11:20:57.276791645Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred alert (3) alert [svc] FrontendPathMedianLatency fired 10:18:15 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-25T11:04:14Z Ended: 2026-08-25T11:37:19Z Severity: Sev3 Labels: alertname="FrontendPathMedianLatency", cluster="ci01-j2151936-svc", component="frontend", method="put", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning" Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j2151936-svc). alert [hcp] KubePodNotReady fired 10:18:15 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 133 time(s) (48 unknown, 85 known) Firing 1: State: Resolved Started: 2026-08-25T11:12:39Z Ended: 2026-08-25T11:19:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig5p67f664gdu1m2g9bl8mcvifgd7-b0n9x4p4j4r7o9p", pod="cluster-network-operator-66bccc674-f9jd2", severity="warning" Description: Pod ocm-arohcpci01-2sdig5p67f664gdu1m2g9bl8mcvifgd7-b0n9x4p4j4r7o9p/cluster-network-operator-66bccc674-f9jd2 has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-08-25T11:12:39Z Ended: 2026-08-25T11:18:48Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigdkfodll0bdu2u3blkkpq2isdp01-ea-listqqp748", pod="etcd-2", severity="warning" Description: Pod ocm-arohcpci01-2sdigdkfodll0bdu2u3blkkpq2isdp01-ea-listqqp748/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-08-25T11:14:44Z Ended: 2026-08-25T11:21:41Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig34vqir757268i5b4b1ol83t92ql-f3e5u5i7x7y2v3b", pod="dns-operator-68d4b6f668-l9ddx", severity="warning" Description: Pod ocm-arohcpci01-2sdig34vqir757268i5b4b1ol83t92ql-f3e5u5i7x7y2v3b/dns-operator-68d4b6f668-l9ddx has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-08-25T11:14:44Z Ended: 2026-08-25T11:21:42Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig8dm0oe272dqghvft882nvmv2jtt-cilium-cl", pod="cluster-network-operator-5765bb4777-txdkz", severity="warning" Description: Pod ocm-arohcpci01-2sdig8dm0oe272dqghvft882nvmv2jtt-cilium-cl/cluster-network-operator-5765bb4777-txdkz has been in a non-ready state for longer than 5 minutes. Firing 5: State: Resolved Started: 2026-08-25T11:14:44Z Ended: 2026-08-25T11:20:41Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig8cujb2061821rrqkcioldosjt2v-g9n3s2f2k8n1t7q", pod="openshift-apiserver-5576f4f4d4-97286", severity="warning" Description: Pod ocm-arohcpci01-2sdig8cujb2061821rrqkcioldosjt2v-g9n3s2f2k8n1t7q/openshift-apiserver-5576f4f4d4-97286 has been in a non-ready state for longer than 5 minutes. Firing 6: State: Resolved Started: 2026-08-25T11:14:44Z Ended: 2026-08-25T11:21:42Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdiftinbamjaq0j5tbanqtu5nns4e4p-t4b7e0d9q4i1q1n", pod="etcd-0", severity="warning" Description: Pod ocm-arohcpci01-2sdiftinbamjaq0j5tbanqtu5nns4e4p-t4b7e0d9q4i1q1n/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 7: State: Resolved Started: 2026-08-25T11:14:44Z Ended: 2026-08-25T11:21:41Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig38ddbi8bj330ci2r2kcpqdbjvuu-v1e7o9p8l0z2o9s", pod="etcd-0", severity="warning" Description: Pod ocm-arohcpci01-2sdig38ddbi8bj330ci2r2kcpqdbjvuu-v1e7o9p8l0z2o9s/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 8: State: Resolved Started: 2026-08-25T11:14:44Z Ended: 2026-08-25T11:20:40Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigarcqk1jc7jjqkmj3beg8b4voupr-o5u3a3y3t1g7p9i", pod="openshift-apiserver-7cdd6bc555-5hzp7", severity="warning" Description: Pod ocm-arohcpci01-2sdigarcqk1jc7jjqkmj3beg8b4voupr-o5u3a3y3t1g7p9i/openshift-apiserver-7cdd6bc555-5hzp7 has been in a non-ready state for longer than 5 minutes. Firing 9: State: Resolved Started: 2026-08-25T11:14:44Z Ended: 2026-08-25T11:21:41Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig819rv81qsapl7h6p95729fbi8l7-u8g6a6c8k9v4d4v", pod="etcd-2", severity="warning" Description: Pod ocm-arohcpci01-2sdig819rv81qsapl7h6p95729fbi8l7-u8g6a6c8k9v4d4v/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 10: State: Resolved Started: 2026-08-25T11:14:44Z Ended: 2026-08-25T11:20:41Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig8tmoe0ak1l116rq7u8af74ia3e6-oidc-wi-cluster", pod="ignition-server-proxy-5bb6484f8f-4797l", severity="warning" Description: Pod ocm-arohcpci01-2sdig8tmoe0ak1l116rq7u8af74ia3e6-oidc-wi-cluster/ignition-server-proxy-5bb6484f8f-4797l has been in a non-ready state for longer than 5 minutes. Firing 11: State: Resolved Started: 2026-08-25T11:14:44Z Ended: 2026-08-25T11:20:41Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig84uregpbaeku352dmjngjarq0l3-z2z7s9l1f1s1q8t", pod="ignition-server-proxy-5d884fd766-zn7rk", severity="warning" Description: Pod ocm-arohcpci01-2sdig84uregpbaeku352dmjngjarq0l3-z2z7s9l1f1s1q8t/ignition-server-proxy-5d884fd766-zn7rk has been in a non-ready state for longer than 5 minutes. Firing 12: State: Resolved Started: 2026-08-25T11:14:44Z Ended: 2026-08-25T11:21:42Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig34vqir757268i5b4b1ol83t92ql-f3e5u5i7x7y2v3b", pod="cluster-node-tuning-operator-688f489b78-msrr5", severity="warning" Description: Pod ocm-arohcpci01-2sdig34vqir757268i5b4b1ol83t92ql-f3e5u5i7x7y2v3b/cluster-node-tuning-operator-688f489b78-msrr5 has been in a non-ready state for longer than 5 minutes. Firing 13: State: Resolved Started: 2026-08-25T11:15:40Z Ended: 2026-08-25T11:21:41Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig8ts4tcq3j83brb5tgspj33kr4jf-s1f3w3v0x3p9g4v", pod="multus-admission-controller-84f697989b-vlk62", severity="warning" Description: Pod ocm-arohcpci01-2sdig8ts4tcq3j83brb5tgspj33kr4jf-s1f3w3v0x3p9g4v/multus-admission-controller-84f697989b-vlk62 has been in a non-ready state for longer than 5 minutes. Firing 14: State: Resolved Started: 2026-08-25T11:15:40Z Ended: 2026-08-25T11:21:41Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig644i6o0am3lvqru4vdu72s9o4kl-w4i4n4y2w1o0u1r", pod="ingress-operator-94d4b54f6-jccsg", severity="warning" Description: Pod ocm-arohcpci01-2sdig644i6o0am3lvqru4vdu72s9o4kl-w4i4n4y2w1o0u1r/ingress-operator-94d4b54f6-jccsg has been in a non-ready state for longer than 5 minutes. Firing 15: State: Resolved Started: 2026-08-25T11:15:40Z Ended: 2026-08-25T11:21:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdiggk1koknr2fnqm2kiv8hqodjkdro-j5z4w7m2q6y2f3e", pod="openshift-controller-manager-787d4c48d9-zmvmp", severity="warning" Description: Pod ocm-arohcpci01-2sdiggk1koknr2fnqm2kiv8hqodjkdro-j5z4w7m2q6y2f3e/openshift-controller-manager-787d4c48d9-zmvmp has been in a non-ready state for longer than 5 minutes. Firing 16: State: Resolved Started: 2026-08-25T11:15:42Z Ended: 2026-08-25T11:21:41Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig8r5a6pfu7k927miu129sk7s4ptu-ephemeral-disk", pod="kube-apiserver-7bcb44bd57-4lfgq", severity="warning" Description: Pod ocm-arohcpci01-2sdig8r5a6pfu7k927miu129sk7s4ptu-ephemeral-disk/kube-apiserver-7bcb44bd57-4lfgq has been in a non-ready state for longer than 5 minutes. Firing 17: State: Resolved Started: 2026-08-25T11:15:42Z Ended: 2026-08-25T11:21:42Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig644i6o0am3lvqru4vdu72s9o4kl-w4i4n4y2w1o0u1r", pod="cluster-network-operator-6969665b4-vvwq5", severity="warning" Description: Pod ocm-arohcpci01-2sdig644i6o0am3lvqru4vdu72s9o4kl-w4i4n4y2w1o0u1r/cluster-network-operator-6969665b4-vvwq5 has been in a non-ready state for longer than 5 minutes. Firing 18: State: Resolved Started: 2026-08-25T11:15:43Z Ended: 2026-08-25T11:29:42Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig8cujb2061821rrqkcioldosjt2v-g9n3s2f2k8n1t7q", pod="router-55674ccc44-pvvww", severity="warning" Description: Pod ocm-arohcpci01-2sdig8cujb2061821rrqkcioldosjt2v-g9n3s2f2k8n1t7q/router-55674ccc44-pvvww has been in a non-ready state for longer than 5 minutes. Firing 19: State: Resolved Started: 2026-08-25T11:16:40Z Ended: 2026-08-25T11:34:40Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigaotum572v7tp065fldok2ga0duj-v20260901", pod="router-dc6b8f666-nfl8q", severity="warning" Description: Pod ocm-arohcpci01-2sdigaotum572v7tp065fldok2ga0duj-v20260901/router-dc6b8f666-nfl8q has been in a non-ready state for longer than 5 minutes. Firing 20: State: Resolved Started: 2026-08-25T11:16:41Z Ended: 2026-08-25T11:23:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigapoi3212giljien9ra888at6kjb-sre-hcp-cluster", pod="ingress-operator-788df6f5bf-hkq6p", severity="warning" Description: Pod ocm-arohcpci01-2sdigapoi3212giljien9ra888at6kjb-sre-hcp-cluster/ingress-operator-788df6f5bf-hkq6p has been in a non-ready state for longer than 5 minutes. Firing 21: State: Resolved Started: 2026-08-25T11:16:41Z Ended: 2026-08-25T11:29:42Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig3erhiqelo66rmekapehp0qpnvp5-u7i2f6t6c4k3k0m", pod="router-85b5dfdd44-98b8g", severity="warning" Description: Pod ocm-arohcpci01-2sdig3erhiqelo66rmekapehp0qpnvp5-u7i2f6t6c4k3k0m/router-85b5dfdd44-98b8g has been in a non-ready state for longer than 5 minutes. Firing 22: State: Resolved Started: 2026-08-25T11:16:41Z Ended: 2026-08-25T11:22:41Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig38eme5vr4f34ngc821ehqehdv26-h3q7l3s0e8q5p1m", pod="azure-file-csi-driver-controller-6445ccdb57-69cxn", severity="warning" Description: Pod ocm-arohcpci01-2sdig38eme5vr4f34ngc821ehqehdv26-h3q7l3s0e8q5p1m/azure-file-csi-driver-controller-6445ccdb57-69cxn has been in a non-ready state for longer than 5 minutes. Firing 23: State: Resolved Started: 2026-08-25T11:16:41Z Ended: 2026-08-25T11:22:41Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig5v5uqol5cmac2j0cndb1s9cu2tq-i8j6t1e4v0q3h0a", pod="cluster-network-operator-86c4c9bf7c-2qtjg", severity="warning" Description: Pod ocm-arohcpci01-2sdig5v5uqol5cmac2j0cndb1s9cu2tq-i8j6t1e4v0q3h0a/cluster-network-operator-86c4c9bf7c-2qtjg has been in a non-ready state for longer than 5 minutes. Firing 24: State: Resolved Started: 2026-08-25T11:17:41Z Ended: 2026-08-25T11:24:42Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig5vcmg991o3cj5ur28dqke5597h9-h5a5a9y0a8v5l4j", pod="cluster-network-operator-67d5d5b7d6-96ptn", severity="warning" Description: Pod ocm-arohcpci01-2sdig5vcmg991o3cj5ur28dqke5597h9-h5a5a9y0a8v5l4j/cluster-network-operator-67d5d5b7d6-96ptn has been in a non-ready state for longer than 5 minutes. Firing 25: State: Resolved Started: 2026-08-25T11:18:42Z Ended: 2026-08-25T11:24:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig8tmoe0ak1l116rq7u8af74ia3e6-oidc-wi-cluster", pod="ingress-operator-68d4cc6849-tq4gb", severity="warning" Description: Pod ocm-arohcpci01-2sdig8tmoe0ak1l116rq7u8af74ia3e6-oidc-wi-cluster/ingress-operator-68d4cc6849-tq4gb has been in a non-ready state for longer than 5 minutes. Firing 26: State: Resolved Started: 2026-08-25T11:18:42Z Ended: 2026-08-25T11:24:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig3erhiqelo66rmekapehp0qpnvp5-u7i2f6t6c4k3k0m", pod="machine-approver-64b9b94cb4-ltnvw", severity="warning" Description: Pod ocm-arohcpci01-2sdig3erhiqelo66rmekapehp0qpnvp5-u7i2f6t6c4k3k0m/machine-approver-64b9b94cb4-ltnvw has been in a non-ready state for longer than 5 minutes. Firing 27: State: Resolved Started: 2026-08-25T11:18:42Z Ended: 2026-08-25T11:31:39Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig862dk1tnff6ftg9i4f840029fum-j6u3d4g4y5d4a5y", pod="router-dc7df8b4b-mb5bt", severity="warning" Description: Pod ocm-arohcpci01-2sdig862dk1tnff6ftg9i4f840029fum-j6u3d4g4y5d4a5y/router-dc7df8b4b-mb5bt has been in a non-ready state for longer than 5 minutes. Firing 28: State: Resolved Started: 2026-08-25T11:18:42Z Ended: 2026-08-25T11:24:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig8tmoe0ak1l116rq7u8af74ia3e6-oidc-wi-cluster", pod="cluster-image-registry-operator-57d6dcf4d4-sp59g", severity="warning" Description: Pod ocm-arohcpci01-2sdig8tmoe0ak1l116rq7u8af74ia3e6-oidc-wi-cluster/cluster-image-registry-operator-57d6dcf4d4-sp59g has been in a non-ready state for longer than 5 minutes. Firing 29: State: Resolved Started: 2026-08-25T11:18:42Z Ended: 2026-08-25T11:24:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigana8st2onnrv326q7o7860hbrtm-z2n7y5t7o2w2p0q", pod="ingress-operator-57d759d85c-cjhvx", severity="warning" Description: Pod ocm-arohcpci01-2sdigana8st2onnrv326q7o7860hbrtm-z2n7y5t7o2w2p0q/ingress-operator-57d759d85c-cjhvx has been in a non-ready state for longer than 5 minutes. Firing 30: State: Resolved Started: 2026-08-25T11:18:42Z Ended: 2026-08-25T11:24:42Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigana8st2onnrv326q7o7860hbrtm-z2n7y5t7o2w2p0q", pod="dns-operator-55657bf974-bnrbh", severity="warning" Description: Pod ocm-arohcpci01-2sdigana8st2onnrv326q7o7860hbrtm-z2n7y5t7o2w2p0q/dns-operator-55657bf974-bnrbh has been in a non-ready state for longer than 5 minutes. Firing 31: State: Resolved Started: 2026-08-25T11:18:42Z Ended: 2026-08-25T11:24:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig608melq35so7gm1quh13emo8muf-q7j3f1e7u4f3t7q", pod="cluster-network-operator-7bfd8bf4bc-2wwlx", severity="warning" Description: Pod ocm-arohcpci01-2sdig608melq35so7gm1quh13emo8muf-q7j3f1e7u4f3t7q/cluster-network-operator-7bfd8bf4bc-2wwlx has been in a non-ready state for longer than 5 minutes. Firing 32: State: Resolved Started: 2026-08-25T11:18:42Z Ended: 2026-08-25T11:24:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigana8st2onnrv326q7o7860hbrtm-z2n7y5t7o2w2p0q", pod="cluster-network-operator-6f59dc5797-8l6qs", severity="warning" Description: Pod ocm-arohcpci01-2sdigana8st2onnrv326q7o7860hbrtm-z2n7y5t7o2w2p0q/cluster-network-operator-6f59dc5797-8l6qs has been in a non-ready state for longer than 5 minutes. Firing 33: State: Resolved Started: 2026-08-25T11:18:42Z Ended: 2026-08-25T11:24:42Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig8tmoe0ak1l116rq7u8af74ia3e6-oidc-wi-cluster", pod="openshift-controller-manager-758df4ffd7-r9m45", severity="warning" Description: Pod ocm-arohcpci01-2sdig8tmoe0ak1l116rq7u8af74ia3e6-oidc-wi-cluster/openshift-controller-manager-758df4ffd7-r9m45 has been in a non-ready state for longer than 5 minutes. Firing 34: State: Resolved Started: 2026-08-25T11:18:42Z Ended: 2026-08-25T11:24:42Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigana8st2onnrv326q7o7860hbrtm-z2n7y5t7o2w2p0q", pod="dns-operator-7d75777d6b-v8zxj", severity="warning" Description: Pod ocm-arohcpci01-2sdigana8st2onnrv326q7o7860hbrtm-z2n7y5t7o2w2p0q/dns-operator-7d75777d6b-v8zxj has been in a non-ready state for longer than 5 minutes. Firing 35: State: Resolved Started: 2026-08-25T11:19:42Z Ended: 2026-08-25T11:31:40Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig38eme5vr4f34ngc821ehqehdv26-h3q7l3s0e8q5p1m", pod="azure-disk-csi-driver-controller-fcf46799f-qp4v6", severity="warning" Description: Pod ocm-arohcpci01-2sdig38eme5vr4f34ngc821ehqehdv26-h3q7l3s0e8q5p1m/azure-disk-csi-driver-controller-fcf46799f-qp4v6 has been in a non-ready state for longer than 5 minutes. Firing 36: State: Resolved Started: 2026-08-25T11:19:42Z Ended: 2026-08-25T11:26:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig8tjgu9c5er3bs4kdcuvmp48h9bj-w6j2p6x4l5f8m1p", pod="csi-snapshot-controller-operator-79cfccd995-9zlzj", severity="warning" Description: Pod ocm-arohcpci01-2sdig8tjgu9c5er3bs4kdcuvmp48h9bj-w6j2p6x4l5f8m1p/csi-snapshot-controller-operator-79cfccd995-9zlzj has been in a non-ready state for longer than 5 minutes. Firing 37: State: Resolved Started: 2026-08-25T11:19:42Z Ended: 2026-08-25T11:25:41Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig8mc00gu1gpaopm9r03966vng1f9-d1b4g5o4p9w4n0l", pod="multus-admission-controller-79d7ccc4c8-rrxfn", severity="warning" Description: Pod ocm-arohcpci01-2sdig8mc00gu1gpaopm9r03966vng1f9-d1b4g5o4p9w4n0l/multus-admission-controller-79d7ccc4c8-rrxfn has been in a non-ready state for longer than 5 minutes. Firing 38: State: Resolved Started: 2026-08-25T11:19:42Z Ended: 2026-08-25T11:25:41Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig5v5uqol5cmac2j0cndb1s9cu2tq-i8j6t1e4v0q3h0a", pod="azure-file-csi-driver-operator-678d98dc57-zb8kj", severity="warning" Description: Pod ocm-arohcpci01-2sdig5v5uqol5cmac2j0cndb1s9cu2tq-i8j6t1e4v0q3h0a/azure-file-csi-driver-operator-678d98dc57-zb8kj has been in a non-ready state for longer than 5 minutes. Firing 39: State: Resolved Started: 2026-08-25T11:19:42Z Ended: 2026-08-25T11:25:41Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihc38ad9t2fqbgqsetulikg35kj4d-agg-cluster", pod="openshift-apiserver-6b856db4fd-t2hdh", severity="warning" Description: Pod ocm-arohcpci01-2sdihc38ad9t2fqbgqsetulikg35kj4d-agg-cluster/openshift-apiserver-6b856db4fd-t2hdh has been in a non-ready state for longer than 5 minutes. Firing 40: State: Resolved Started: 2026-08-25T11:19:42Z Ended: 2026-08-25T11:25:41Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig5oudr0ppt5j63nvg2lathk0pvcd-b9e4d1i8o2d4s4b", pod="openshift-controller-manager-547df955b4-g2v7j", severity="warning" Description: Pod ocm-arohcpci01-2sdig5oudr0ppt5j63nvg2lathk0pvcd-b9e4d1i8o2d4s4b/openshift-controller-manager-547df955b4-g2v7j has been in a non-ready state for longer than 5 minutes. Firing 41: State: Resolved Started: 2026-08-25T11:19:42Z Ended: 2026-08-25T11:29:42Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig8tjgu9c5er3bs4kdcuvmp48h9bj-w6j2p6x4l5f8m1p", pod="ingress-operator-5c7c7d9d95-dljws", severity="warning" Description: Pod ocm-arohcpci01-2sdig8tjgu9c5er3bs4kdcuvmp48h9bj-w6j2p6x4l5f8m1p/ingress-operator-5c7c7d9d95-dljws has been in a non-ready state for longer than 5 minutes. Firing 42: State: Resolved Started: 2026-08-25T11:19:42Z Ended: 2026-08-25T11:25:41Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigapoi3212giljien9ra888at6kjb-sre-hcp-cluster", pod="azure-disk-csi-driver-operator-5d7595c449-nrwjg", severity="warning" Description: Pod ocm-arohcpci01-2sdigapoi3212giljien9ra888at6kjb-sre-hcp-cluster/azure-disk-csi-driver-operator-5d7595c449-nrwjg has been in a non-ready state for longer than 5 minutes. Firing 43: State: Resolved Started: 2026-08-25T11:19:42Z Ended: 2026-08-25T11:25:41Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig8mc00gu1gpaopm9r03966vng1f9-d1b4g5o4p9w4n0l", pod="multus-admission-controller-79d7ccc4c8-w86gx", severity="warning" Description: Pod ocm-arohcpci01-2sdig8mc00gu1gpaopm9r03966vng1f9-d1b4g5o4p9w4n0l/multus-admission-controller-79d7ccc4c8-w86gx has been in a non-ready state for longer than 5 minutes. Firing 44: State: Resolved Started: 2026-08-25T11:20:40Z Ended: 2026-08-25T11:26:41Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigau4l6s2auv43thna4dh8btjadmo-private-ingress", pod="machine-approver-7ccbff6cdc-qhdv8", severity="warning" Description: Pod ocm-arohcpci01-2sdigau4l6s2auv43thna4dh8btjadmo-private-ingress/machine-approver-7ccbff6cdc-qhdv8 has been in a non-ready state for longer than 5 minutes. Firing 45: State: Resolved Started: 2026-08-25T11:20:40Z Ended: 2026-08-25T11:28:40Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig8tjgu9c5er3bs4kdcuvmp48h9bj-w6j2p6x4l5f8m1p", pod="cluster-network-operator-5d8f449bdd-skx5k", severity="warning" Description: Pod ocm-arohcpci01-2sdig8tjgu9c5er3bs4kdcuvmp48h9bj-w6j2p6x4l5f8m1p/cluster-network-operator-5d8f449bdd-skx5k has been in a non-ready state for longer than 5 minutes. Firing 46: State: Resolved Started: 2026-08-25T11:20:40Z Ended: 2026-08-25T11:26:41Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig8tjgu9c5er3bs4kdcuvmp48h9bj-w6j2p6x4l5f8m1p", pod="cluster-storage-operator-c9f4b55c6-sgsj5", severity="warning" Description: Pod ocm-arohcpci01-2sdig8tjgu9c5er3bs4kdcuvmp48h9bj-w6j2p6x4l5f8m1p/cluster-storage-operator-c9f4b55c6-sgsj5 has been in a non-ready state for longer than 5 minutes. Firing 47: State: Resolved Started: 2026-08-25T11:20:40Z Ended: 2026-08-25T11:26:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig8tjgu9c5er3bs4kdcuvmp48h9bj-w6j2p6x4l5f8m1p", pod="dns-operator-58f6959675-w8npm", severity="warning" Description: Pod ocm-arohcpci01-2sdig8tjgu9c5er3bs4kdcuvmp48h9bj-w6j2p6x4l5f8m1p/dns-operator-58f6959675-w8npm has been in a non-ready state for longer than 5 minutes. Firing 48: State: Resolved Started: 2026-08-25T11:20:40Z Ended: 2026-08-25T11:26:42Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2151936-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig8tjgu9c5er3bs4kdcuvmp48h9bj-w6j2p6x4l5f8m1p", pod="cluster-version-operator-64767fc9fc-j6c29", severity="warning" Description: Pod ocm-arohcpci01-2sdig8tjgu9c5er3bs4kdcuvmp48h9bj-w6j2p6x4l5f8m1p/cluster-version-operator-64767fc9fc-j6c29 has been in a non-ready state for longer than 5 minutes. alert [svc] UJClusterProvisionErrors6h30m fired 10:18:15 UTC · failed at alert · test [aro-hcp-observability] [svc] alert UJClusterProvisionErrors6h30m does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-08-25T11:51:56Z Severity: Sev3 Labels: alertname="UJClusterProvisionErrors6h30m", cluster="ci01-j2151936-svc", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2151936/providers/microsoft.monitor/accounts/services-j2151936", severity="3", short_window="30m", slo="cluster-provision-errors" Description: More than 30% of cluster create (install) operations are in failed state sustained over 30 minutes, indicating a medium error budget burn (6x) that would exhaust the 95% SLO budget in ~28 hours.
10:03:11	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 515e34b	provision	failed	⚠#6583 (open)	1	provision (1) ERROR: The subscription of 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' doesn't exist in cloud 'AzureCloud'. 10:03:11 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/swift-vnet (stamp=1) · suite step graph Show raw failure time=2026-08-25T10:31:54.607Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=swift-vnet stamp=1 description="Step swift-vnet\n Kind: Shell\n Command: ./swift-vnet.sh\n" time=2026-08-25T10:31:55.356Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=swift-vnet stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: Failed to load or parse file /tmp/azure-cli-config-3697365572/azureProfile.json. It will be overridden by default settings.\nERROR: The subscription of 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' doesn't exist in cloud 'AzureCloud'.\n exit status 1"
10:03:03	pull-ci-Azure-ARO-HCP-main-e2e-parallel head bf87cfe	alert, e2e	failed	#6658 (open)	7	e2e (6) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 10:03:03 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc000e8d5e0>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-s7knrjd2n4p9" to finish creating: GET https://rp.j8682368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a20d1a46-4016-4993-942c-628985f40d60 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a20d1a46-4016-4993-942c-628985f40d60", "name": "a20d1a46-4016-4993-942c-628985f40d60", "status": "Failed", "startTime": "2026-08-25T10:52:09.390785108Z", "endTime": "2026-08-25T11:11:13.084035599Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc000e8d5e0>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-s7knrjd2n4p9" to finish creating: GET https://rp.j8682368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a20d1a46-4016-4993-942c-628985f40d60 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a20d1a46-4016-4993-942c-628985f40d60", "name": "a20d1a46-4016-4993-942c-628985f40d60", "status": "Failed", "startTime": "2026-08-25T10:52:09.390785108Z", "endTime": "2026-08-25T11:11:13.084035599Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 10:03:03 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:192]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster" Unexpected error: <errors.joinError \| 0xc0012ca6a8>: VerifyNodesReady failed: there are not ready nodes: [cilium-cluster-cilium-np-vknrp-c5ljh cilium-cluster-cilium-np-vknrp-thjfc] failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-9jxg574dgjtb" to finish creating: GET https://rp.j8682368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/df94ac92-d0eb-418f-ba1a-c79aa21c8ec3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/df94ac92-d0eb-418f-ba1a-c79aa21c8ec3", "name": "df94ac92-d0eb-418f-ba1a-c79aa21c8ec3", "status": "Failed", "startTime": "2026-08-25T11:06:59.904255754Z", "endTime": "2026-08-25T11:25:59.419108691Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:192]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster" Unexpected error: <errors.joinError \| 0xc0012ca6a8>: VerifyNodesReady failed: there are not ready nodes: [cilium-cluster-cilium-np-vknrp-c5ljh cilium-cluster-cilium-np-vknrp-thjfc] failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-9jxg574dgjtb" to finish creating: GET https://rp.j8682368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/df94ac92-d0eb-418f-ba1a-c79aa21c8ec3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/df94ac92-d0eb-418f-ba1a-c79aa21c8ec3", "name": "df94ac92-d0eb-418f-ba1a-c79aa21c8ec3", "status": "Failed", "startTime": "2026-08-25T11:06:59.904255754Z", "endTime": "2026-08-25T11:25:59.419108691Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 10:03:03 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc0010f88c0>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-kd8kq878wflk" to finish creating: GET https://rp.j8682368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/70626b07-0cb8-4f0d-933a-2f9445223f6a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/70626b07-0cb8-4f0d-933a-2f9445223f6a", "name": "70626b07-0cb8-4f0d-933a-2f9445223f6a", "status": "Failed", "startTime": "2026-08-25T10:52:07.614945455Z", "endTime": "2026-08-25T11:11:13.090308867Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc0010f88c0>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-kd8kq878wflk" to finish creating: GET https://rp.j8682368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/70626b07-0cb8-4f0d-933a-2f9445223f6a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/70626b07-0cb8-4f0d-933a-2f9445223f6a", "name": "70626b07-0cb8-4f0d-933a-2f9445223f6a", "status": "Failed", "startTime": "2026-08-25T10:52:07.614945455Z", "endTime": "2026-08-25T11:11:13.090308867Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 10:03:03 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-jjpphbklnmpm/cluster-candidate-5-0-8jxb6p should provision Unexpected error: <fmt.wrapError \| 0xc001a56220>: failed to create HCP cluster cluster-candidate-5-0-8jxb6p: failed waiting for cluster="cluster-candidate-5-0-8jxb6p" in resourcegroup="rg-candidate-5-0-jjpphbklnmpm" to finish creating: GET https://rp.j8682368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/66297db9-5baf-4aaa-8975-b676c020ea95 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/66297db9-5baf-4aaa-8975-b676c020ea95", "name": "66297db9-5baf-4aaa-8975-b676c020ea95", "status": "Failed", "startTime": "2026-08-25T10:52:17.485728635Z", "endTime": "2026-08-25T11:11:23.092877103Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-jjpphbklnmpm/cluster-candidate-5-0-8jxb6p should provision Unexpected error: <fmt.wrapError \| 0xc001a56220>: failed to create HCP cluster cluster-candidate-5-0-8jxb6p: failed waiting for cluster="cluster-candidate-5-0-8jxb6p" in resourcegroup="rg-candidate-5-0-jjpphbklnmpm" to finish creating: GET https://rp.j8682368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/66297db9-5baf-4aaa-8975-b676c020ea95 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/66297db9-5baf-4aaa-8975-b676c020ea95", "name": "66297db9-5baf-4aaa-8975-b676c020ea95", "status": "Failed", "startTime": "2026-08-25T10:52:17.485728635Z", "endTime": "2026-08-25T11:11:23.092877103Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 10:03:03 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:111]: failed to create node pool "np-5cwprg" for cluster "cluster-hs-pre-5cwprg" Unexpected error: <fmt.wrapError \| 0xc0010169e0>: failed to create NodePool np-5cwprg: failed waiting for nodepool="np-5cwprg" for cluster "cluster-hs-pre-5cwprg" in resourcegroup="rg-hs-pre--lqnv4n95tx47" to finish creating: GET https://rp.j8682368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/40364dda-e147-4d07-941d-26fe941e32ab -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/40364dda-e147-4d07-941d-26fe941e32ab", "name": "40364dda-e147-4d07-941d-26fe941e32ab", "status": "Failed", "startTime": "2026-08-25T11:07:14.906937807Z", "endTime": "2026-08-25T11:26:19.497689075Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:111]: failed to create node pool "np-5cwprg" for cluster "cluster-hs-pre-5cwprg" Unexpected error: <fmt.wrapError \| 0xc0010169e0>: failed to create NodePool np-5cwprg: failed waiting for nodepool="np-5cwprg" for cluster "cluster-hs-pre-5cwprg" in resourcegroup="rg-hs-pre--lqnv4n95tx47" to finish creating: GET https://rp.j8682368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/40364dda-e147-4d07-941d-26fe941e32ab -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/40364dda-e147-4d07-941d-26fe941e32ab", "name": "40364dda-e147-4d07-941d-26fe941e32ab", "status": "Failed", "startTime": "2026-08-25T11:07:14.906937807Z", "endTime": "2026-08-25T11:26:19.497689075Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 10:03:03 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1 Unexpected error: <fmt.wrapError \| 0xc0015281e0>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-fmb6zrrk7zsv" to finish creating: GET https://rp.j8682368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3f451991-d984-48c4-98fe-f9cd0fb13d1a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3f451991-d984-48c4-98fe-f9cd0fb13d1a", "name": "3f451991-d984-48c4-98fe-f9cd0fb13d1a", "status": "Failed", "startTime": "2026-08-25T11:07:18.751416822Z", "endTime": "2026-08-25T11:26:19.532469118Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1 Unexpected error: <fmt.wrapError \| 0xc0015281e0>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-fmb6zrrk7zsv" to finish creating: GET https://rp.j8682368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3f451991-d984-48c4-98fe-f9cd0fb13d1a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3f451991-d984-48c4-98fe-f9cd0fb13d1a", "name": "3f451991-d984-48c4-98fe-f9cd0fb13d1a", "status": "Failed", "startTime": "2026-08-25T11:07:18.751416822Z", "endTime": "2026-08-25T11:26:19.532469118Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 10:03:03 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-25T11:04:49Z Ended: 2026-08-25T11:19:51Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8682368-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdibolp1edv6hvnhp9i6alddcpmqbrg-x1v3s6y8u0i9u9q", pod="router-7dc45bdf49-z52rd", severity="warning" Description: Pod ocm-arohcpci01-2sdibolp1edv6hvnhp9i6alddcpmqbrg-x1v3s6y8u0i9u9q/router-7dc45bdf49-z52rd has been in a non-ready state for longer than 5 minutes.
10:03:00	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 87c9d3c	alert, e2e	failed	⚠#6662 (open)	52	e2e (49) ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:442]: failed to create HCP cluster np-skip-minor-g2j9ht at version 4.22.11 Unexpected error: <fmt.wrapError \| 0xc001c2e9e0>: failed to create HCP cluster np-skip-minor-g2j9ht: failed waiting for cluster="np-skip-minor-g2j9ht" in resourcegroup="rg-np-skip-minor-g2j9ht-c89fhbmn6zb7" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/104bf35c-3c35-49bd-a295-bd89c97479d4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/104bf35c-3c35-49bd-a295-bd89c97479d4", "name": "104bf35c-3c35-49bd-a295-bd89c97479d4", "status": "Failed", "startTime": "2026-08-25T11:05:08.489918597Z", "endTime": "2026-08-25T11:24:17.594649285Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, redhat-operators-catalog, cluster-version-operator, cluster-storage-operator, packageserver, cluster-network-operator, dns-operator, certified-operators-catalog, hosted-cluster-config-operator, csi-snapshot-controller-operator, catalog-operator, olm-operator, community-operators-catalog, ingress-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 2 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:442]: failed to create HCP cluster np-skip-minor-g2j9ht at version 4.22.11 Unexpected error: <fmt.wrapError \| 0xc001c2e9e0>: failed to create HCP cluster np-skip-minor-g2j9ht: failed waiting for cluster="np-skip-minor-g2j9ht" in resourcegroup="rg-np-skip-minor-g2j9ht-c89fhbmn6zb7" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/104bf35c-3c35-49bd-a295-bd89c97479d4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/104bf35c-3c35-49bd-a295-bd89c97479d4", "name": "104bf35c-3c35-49bd-a295-bd89c97479d4", "status": "Failed", "startTime": "2026-08-25T11:05:08.489918597Z", "endTime": "2026-08-25T11:24:17.594649285Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, redhat-operators-catalog, cluster-version-operator, cluster-storage-operator, packageserver, cluster-network-operator, dns-operator, certified-operators-catalog, hosted-cluster-config-operator, csi-snapshot-controller-operator, catalog-operator, olm-operator, community-operators-catalog, ingress-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 2 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-node-tuning-operator, cluster-storage-operator, csi-snapshot-controller-operator, dns-operator, kube-controller-manager, packageserver]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-kp6cj8gslgn7/cluster-candidate-4-20-2ftmsx should provision Unexpected error: <fmt.wrapError \| 0xc00101ebe0>: failed to create HCP cluster cluster-candidate-4-20-2ftmsx: failed waiting for cluster="cluster-candidate-4-20-2ftmsx" in resourcegroup="rg-candidate-4-20-kp6cj8gslgn7" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9ce27661-0cfc-4c0b-a2fe-99358f1c893d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9ce27661-0cfc-4c0b-a2fe-99358f1c893d", "name": "9ce27661-0cfc-4c0b-a2fe-99358f1c893d", "status": "Failed", "startTime": "2026-08-25T11:05:05.525443298Z", "endTime": "2026-08-25T11:24:07.636192222Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator, cluster-network-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, packageserver, cluster-storage-operator; hosted cluster degraded: UnavailableReplicas: [cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 2 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-kp6cj8gslgn7/cluster-candidate-4-20-2ftmsx should provision Unexpected error: <fmt.wrapError \| 0xc00101ebe0>: failed to create HCP cluster cluster-candidate-4-20-2ftmsx: failed waiting for cluster="cluster-candidate-4-20-2ftmsx" in resourcegroup="rg-candidate-4-20-kp6cj8gslgn7" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9ce27661-0cfc-4c0b-a2fe-99358f1c893d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9ce27661-0cfc-4c0b-a2fe-99358f1c893d", "name": "9ce27661-0cfc-4c0b-a2fe-99358f1c893d", "status": "Failed", "startTime": "2026-08-25T11:05:05.525443298Z", "endTime": "2026-08-25T11:24:07.636192222Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator, cluster-network-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, packageserver, cluster-storage-operator; hosted cluster degraded: UnavailableReplicas: [cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 2 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: WaitingForAvailable: Waiting for Kube APIServer deployment to become available; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-sjzzj9kjw9kf/cluster-candidate-5-0-rjlgtk should provision Unexpected error: <fmt.wrapError \| 0xc000f3a660>: failed to create HCP cluster cluster-candidate-5-0-rjlgtk: failed waiting for cluster="cluster-candidate-5-0-rjlgtk" in resourcegroup="rg-candidate-5-0-sjzzj9kjw9kf" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8dd830d6-347e-4473-bb10-124104e89a73 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8dd830d6-347e-4473-bb10-124104e89a73", "name": "8dd830d6-347e-4473-bb10-124104e89a73", "status": "Failed", "startTime": "2026-08-25T11:04:56.05360308Z", "endTime": "2026-08-25T11:23:57.532994811Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: WaitingForAvailable: Waiting for Kube APIServer deployment to become available; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 2 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-sjzzj9kjw9kf/cluster-candidate-5-0-rjlgtk should provision Unexpected error: <fmt.wrapError \| 0xc000f3a660>: failed to create HCP cluster cluster-candidate-5-0-rjlgtk: failed waiting for cluster="cluster-candidate-5-0-rjlgtk" in resourcegroup="rg-candidate-5-0-sjzzj9kjw9kf" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8dd830d6-347e-4473-bb10-124104e89a73 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8dd830d6-347e-4473-bb10-124104e89a73", "name": "8dd830d6-347e-4473-bb10-124104e89a73", "status": "Failed", "startTime": "2026-08-25T11:04:56.05360308Z", "endTime": "2026-08-25T11:23:57.532994811Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: WaitingForAvailable: Waiting for Kube APIServer deployment to become available; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 2 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator, ignition-server-proxy, kube-controller-manager, openshift-controller-manager, openshift-route-controller-manager]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:92]: failed to create HCP cluster "sre-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc0010003c0>: failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-dgghk7xd8fkc" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/813de9e6-c926-4fa8-a38a-500226e3059a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/813de9e6-c926-4fa8-a38a-500226e3059a", "name": "813de9e6-c926-4fa8-a38a-500226e3059a", "status": "Failed", "startTime": "2026-08-25T11:05:05.708804056Z", "endTime": "2026-08-25T11:24:07.512424839Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, hosted-cluster-config-operator, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:92]: failed to create HCP cluster "sre-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc0010003c0>: failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-dgghk7xd8fkc" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/813de9e6-c926-4fa8-a38a-500226e3059a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/813de9e6-c926-4fa8-a38a-500226e3059a", "name": "813de9e6-c926-4fa8-a38a-500226e3059a", "status": "Failed", "startTime": "2026-08-25T11:05:05.708804056Z", "endTime": "2026-08-25T11:24:07.512424839Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, hosted-cluster-config-operator, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-4gfnzd with version 4.21.30 Unexpected error: <fmt.wrapError \| 0xc000b190c0>: failed to create HCP cluster np-version-upgrade-cluster-4gfnzd: failed waiting for cluster="np-version-upgrade-cluster-4gfnzd" in resourcegroup="rg-np-version-upgrade-4gfnzd-rj8pxrlvvg4z" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d916a34c-5fd9-4150-b24c-9e76f99b04aa -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d916a34c-5fd9-4150-b24c-9e76f99b04aa", "name": "d916a34c-5fd9-4150-b24c-9e76f99b04aa", "status": "Failed", "startTime": "2026-08-25T11:04:56.664010297Z", "endTime": "2026-08-25T11:23:57.583359659Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, openshift-controller-manager, cluster-storage-operator, cluster-network-operator, ingress-operator, olm-operator, packageserver, dns-operator, catalog-operator, csi-snapshot-controller-operator, hosted-cluster-config-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-4gfnzd with version 4.21.30 Unexpected error: <fmt.wrapError \| 0xc000b190c0>: failed to create HCP cluster np-version-upgrade-cluster-4gfnzd: failed waiting for cluster="np-version-upgrade-cluster-4gfnzd" in resourcegroup="rg-np-version-upgrade-4gfnzd-rj8pxrlvvg4z" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d916a34c-5fd9-4150-b24c-9e76f99b04aa -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d916a34c-5fd9-4150-b24c-9e76f99b04aa", "name": "d916a34c-5fd9-4150-b24c-9e76f99b04aa", "status": "Failed", "startTime": "2026-08-25T11:04:56.664010297Z", "endTime": "2026-08-25T11:23:57.583359659Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, openshift-controller-manager, cluster-storage-operator, cluster-network-operator, ingress-operator, olm-operator, packageserver, dns-operator, catalog-operator, csi-snapshot-controller-operator, hosted-cluster-config-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, packageserver]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test Unexpected error: <fmt.wrapError \| 0xc000cd4b20>: failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-7nfq89d68b2j" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6b32b4b1-4670-40ea-8086-f9466f12ce3e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6b32b4b1-4670-40ea-8086-f9466f12ce3e", "name": "6b32b4b1-4670-40ea-8086-f9466f12ce3e", "status": "Failed", "startTime": "2026-08-25T11:05:06.362546319Z", "endTime": "2026-08-25T11:24:07.656848362Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, redhat-operators-catalog, certified-operators-catalog, openshift-controller-manager, cluster-node-tuning-operator, community-operators-catalog, olm-operator, cluster-storage-operator, cluster-version-operator, openshift-route-controller-manager, cluster-network-operator, catalog-operator, redhat-marketplace-catalog, cluster-policy-controller, packageserver, dns-operator, ingress-operator, cluster-image-registry-operator, hosted-cluster-config-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test Unexpected error: <fmt.wrapError \| 0xc000cd4b20>: failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-7nfq89d68b2j" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6b32b4b1-4670-40ea-8086-f9466f12ce3e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6b32b4b1-4670-40ea-8086-f9466f12ce3e", "name": "6b32b4b1-4670-40ea-8086-f9466f12ce3e", "status": "Failed", "startTime": "2026-08-25T11:05:06.362546319Z", "endTime": "2026-08-25T11:24:07.656848362Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, redhat-operators-catalog, certified-operators-catalog, openshift-controller-manager, cluster-node-tuning-operator, community-operators-catalog, olm-operator, cluster-storage-operator, cluster-version-operator, openshift-route-controller-manager, cluster-network-operator, catalog-operator, redhat-marketplace-catalog, cluster-policy-controller, packageserver, dns-operator, ingress-operator, cluster-image-registry-operator, hosted-cluster-config-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [openshift-controller-manager]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster Unexpected error: <fmt.wrapError \| 0xc000976d60>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-sp25f888spq4" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d3198a15-7882-4985-8fb9-601f276da5ee -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d3198a15-7882-4985-8fb9-601f276da5ee", "name": "d3198a15-7882-4985-8fb9-601f276da5ee", "status": "Failed", "startTime": "2026-08-25T11:04:58.802251942Z", "endTime": "2026-08-25T11:24:07.715404645Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: openshift-controller-manager deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster Unexpected error: <fmt.wrapError \| 0xc000976d60>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-sp25f888spq4" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d3198a15-7882-4985-8fb9-601f276da5ee -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d3198a15-7882-4985-8fb9-601f276da5ee", "name": "d3198a15-7882-4985-8fb9-601f276da5ee", "status": "Failed", "startTime": "2026-08-25T11:04:58.802251942Z", "endTime": "2026-08-25T11:24:07.715404645Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: openshift-controller-manager deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0xc001b26780>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-lzwn2n84w4rd" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a49b0ca7-b350-4e35-b5c4-7a6cc066dabb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a49b0ca7-b350-4e35-b5c4-7a6cc066dabb", "name": "a49b0ca7-b350-4e35-b5c4-7a6cc066dabb", "status": "Failed", "startTime": "2026-08-25T11:05:06.912879528Z", "endTime": "2026-08-25T11:24:07.72071407Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0xc001b26780>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-lzwn2n84w4rd" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a49b0ca7-b350-4e35-b5c4-7a6cc066dabb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a49b0ca7-b350-4e35-b5c4-7a6cc066dabb", "name": "a49b0ca7-b350-4e35-b5c4-7a6cc066dabb", "status": "Failed", "startTime": "2026-08-25T11:05:06.912879528Z", "endTime": "2026-08-25T11:24:07.72071407Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0xc000fceaa0>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-4scl6wzd5ltc" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c7f2e105-1507-4d15-a29b-003e59886d0b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c7f2e105-1507-4d15-a29b-003e59886d0b", "name": "c7f2e105-1507-4d15-a29b-003e59886d0b", "status": "Failed", "startTime": "2026-08-25T11:04:46.33753064Z", "endTime": "2026-08-25T11:23:47.467774327Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0xc000fceaa0>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-4scl6wzd5ltc" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c7f2e105-1507-4d15-a29b-003e59886d0b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c7f2e105-1507-4d15-a29b-003e59886d0b", "name": "c7f2e105-1507-4d15-a29b-003e59886d0b", "status": "Failed", "startTime": "2026-08-25T11:04:46.33753064Z", "endTime": "2026-08-25T11:23:47.467774327Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator, ingress-operator, router]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc000d08560>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-mx5c2xs4fbj7" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0554f1ec-f6c0-4f0c-869c-8157fe13f06e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0554f1ec-f6c0-4f0c-869c-8157fe13f06e", "name": "0554f1ec-f6c0-4f0c-869c-8157fe13f06e", "status": "Failed", "startTime": "2026-08-25T11:06:26.179895607Z", "endTime": "2026-08-25T11:25:27.51902962Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc000d08560>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-mx5c2xs4fbj7" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0554f1ec-f6c0-4f0c-869c-8157fe13f06e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0554f1ec-f6c0-4f0c-869c-8157fe13f06e", "name": "0554f1ec-f6c0-4f0c-869c-8157fe13f06e", "status": "Failed", "startTime": "2026-08-25T11:06:26.179895607Z", "endTime": "2026-08-25T11:25:27.51902962Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-x7w9rq with version 4.20.35 Unexpected error: <fmt.wrapError \| 0xc000218ae0>: failed to create HCP cluster np-version-upgrade-cluster-x7w9rq: failed waiting for cluster="np-version-upgrade-cluster-x7w9rq" in resourcegroup="rg-np-version-upgrade-x7w9rq-dvf8r82dxzzq" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/96c551ed-bf22-4ca0-a2f9-b01ce2e4cd5e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/96c551ed-bf22-4ca0-a2f9-b01ce2e4cd5e", "name": "96c551ed-bf22-4ca0-a2f9-b01ce2e4cd5e", "status": "Failed", "startTime": "2026-08-25T11:04:57.577943839Z", "endTime": "2026-08-25T11:23:57.610020208Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: certified-operators-catalog deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-x7w9rq with version 4.20.35 Unexpected error: <fmt.wrapError \| 0xc000218ae0>: failed to create HCP cluster np-version-upgrade-cluster-x7w9rq: failed waiting for cluster="np-version-upgrade-cluster-x7w9rq" in resourcegroup="rg-np-version-upgrade-x7w9rq-dvf8r82dxzzq" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/96c551ed-bf22-4ca0-a2f9-b01ce2e4cd5e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/96c551ed-bf22-4ca0-a2f9-b01ce2e4cd5e", "name": "96c551ed-bf22-4ca0-a2f9-b01ce2e4cd5e", "status": "Failed", "startTime": "2026-08-25T11:04:57.577943839Z", "endTime": "2026-08-25T11:23:57.610020208Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: certified-operators-catalog deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0xc000e84820>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-b9lzcpbpn5wp" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2bbcb592-de1c-4ea4-8806-5849cdf0735a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2bbcb592-de1c-4ea4-8806-5849cdf0735a", "name": "2bbcb592-de1c-4ea4-8806-5849cdf0735a", "status": "Failed", "startTime": "2026-08-25T11:05:06.392903067Z", "endTime": "2026-08-25T11:24:07.668839785Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0xc000e84820>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-b9lzcpbpn5wp" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2bbcb592-de1c-4ea4-8806-5849cdf0735a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2bbcb592-de1c-4ea4-8806-5849cdf0735a", "name": "2bbcb592-de1c-4ea4-8806-5849cdf0735a", "status": "Failed", "startTime": "2026-08-25T11:05:06.392903067Z", "endTime": "2026-08-25T11:24:07.668839785Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-t52ww6" with version 4.22.10 on candidate channel Unexpected error: <fmt.wrapError \| 0xc00183c860>: failed to create HCP cluster cluster-zstream-4-22-t52ww6: failed waiting for cluster="cluster-zstream-4-22-t52ww6" in resourcegroup="rg-zstream-upgrade-4-22-rdnc78mtxx5j" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9c696222-49a6-4ca9-b706-8e5a57435480 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9c696222-49a6-4ca9-b706-8e5a57435480", "name": "9c696222-49a6-4ca9-b706-8e5a57435480", "status": "Failed", "startTime": "2026-08-25T11:05:06.248454811Z", "endTime": "2026-08-25T11:24:07.701920992Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: cluster-image-registry-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-t52ww6" with version 4.22.10 on candidate channel Unexpected error: <fmt.wrapError \| 0xc00183c860>: failed to create HCP cluster cluster-zstream-4-22-t52ww6: failed waiting for cluster="cluster-zstream-4-22-t52ww6" in resourcegroup="rg-zstream-upgrade-4-22-rdnc78mtxx5j" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9c696222-49a6-4ca9-b706-8e5a57435480 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9c696222-49a6-4ca9-b706-8e5a57435480", "name": "9c696222-49a6-4ca9-b706-8e5a57435480", "status": "Failed", "startTime": "2026-08-25T11:05:06.248454811Z", "endTime": "2026-08-25T11:24:07.701920992Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: cluster-image-registry-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-rm8r69s8dhfz/cluster-candidate-4-21-q2f2xq should provision Unexpected error: <fmt.wrapError \| 0xc000902920>: failed to create HCP cluster cluster-candidate-4-21-q2f2xq: failed waiting for cluster="cluster-candidate-4-21-q2f2xq" in resourcegroup="rg-candidate-4-21-rm8r69s8dhfz" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5c21efcc-c66c-4246-a3a7-827947fedab9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5c21efcc-c66c-4246-a3a7-827947fedab9", "name": "5c21efcc-c66c-4246-a3a7-827947fedab9", "status": "Failed", "startTime": "2026-08-25T11:05:06.044151568Z", "endTime": "2026-08-25T11:24:07.669127251Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-rm8r69s8dhfz/cluster-candidate-4-21-q2f2xq should provision Unexpected error: <fmt.wrapError \| 0xc000902920>: failed to create HCP cluster cluster-candidate-4-21-q2f2xq: failed waiting for cluster="cluster-candidate-4-21-q2f2xq" in resourcegroup="rg-candidate-4-21-rm8r69s8dhfz" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5c21efcc-c66c-4246-a3a7-827947fedab9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5c21efcc-c66c-4246-a3a7-827947fedab9", "name": "5c21efcc-c66c-4246-a3a7-827947fedab9", "status": "Failed", "startTime": "2026-08-25T11:05:06.044151568Z", "endTime": "2026-08-25T11:24:07.669127251Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-pgpvxwpm6mjj/cluster-candidate-4-22-bnptgd should provision Unexpected error: <fmt.wrapError \| 0xc001084220>: failed to create HCP cluster cluster-candidate-4-22-bnptgd: failed waiting for cluster="cluster-candidate-4-22-bnptgd" in resourcegroup="rg-candidate-4-22-pgpvxwpm6mjj" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9b420dfc-8b86-46f6-8ff3-d7254009e5e3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9b420dfc-8b86-46f6-8ff3-d7254009e5e3", "name": "9b420dfc-8b86-46f6-8ff3-d7254009e5e3", "status": "Failed", "startTime": "2026-08-25T11:05:05.515128304Z", "endTime": "2026-08-25T11:24:07.660901963Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, community-operators-catalog, cluster-node-tuning-operator, catalog-operator, certified-operators-catalog, hosted-cluster-config-operator, openshift-route-controller-manager, cluster-image-registry-operator, redhat-operators-catalog, cluster-version-operator, cluster-network-operator, dns-operator, ingress-operator, packageserver, openshift-controller-manager, cluster-storage-operator, olm-operator, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-pgpvxwpm6mjj/cluster-candidate-4-22-bnptgd should provision Unexpected error: <fmt.wrapError \| 0xc001084220>: failed to create HCP cluster cluster-candidate-4-22-bnptgd: failed waiting for cluster="cluster-candidate-4-22-bnptgd" in resourcegroup="rg-candidate-4-22-pgpvxwpm6mjj" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9b420dfc-8b86-46f6-8ff3-d7254009e5e3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9b420dfc-8b86-46f6-8ff3-d7254009e5e3", "name": "9b420dfc-8b86-46f6-8ff3-d7254009e5e3", "status": "Failed", "startTime": "2026-08-25T11:05:05.515128304Z", "endTime": "2026-08-25T11:24:07.660901963Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, community-operators-catalog, cluster-node-tuning-operator, catalog-operator, certified-operators-catalog, hosted-cluster-config-operator, openshift-route-controller-manager, cluster-image-registry-operator, redhat-operators-catalog, cluster-version-operator, cluster-network-operator, dns-operator, ingress-operator, packageserver, openshift-controller-manager, cluster-storage-operator, olm-operator, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints Unexpected error: <fmt.wrapError \| 0xc000f3ed80>: failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-klhpfqtd5gz6" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f19d77e4-27f1-47c7-ae51-adcc966d0da7 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f19d77e4-27f1-47c7-ae51-adcc966d0da7", "name": "f19d77e4-27f1-47c7-ae51-adcc966d0da7", "status": "Failed", "startTime": "2026-08-25T11:04:56.024004396Z", "endTime": "2026-08-25T11:23:57.629775108Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints Unexpected error: <fmt.wrapError \| 0xc000f3ed80>: failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-klhpfqtd5gz6" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f19d77e4-27f1-47c7-ae51-adcc966d0da7 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f19d77e4-27f1-47c7-ae51-adcc966d0da7", "name": "f19d77e4-27f1-47c7-ae51-adcc966d0da7", "status": "Failed", "startTime": "2026-08-25T11:04:56.024004396Z", "endTime": "2026-08-25T11:23:57.629775108Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI Unexpected error: <fmt.wrapError \| 0xc0011b8160>: failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-5zct5rkc97n4" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00460c52-4267-4ccc-90ce-ef17247c788e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00460c52-4267-4ccc-90ce-ef17247c788e", "name": "00460c52-4267-4ccc-90ce-ef17247c788e", "status": "Failed", "startTime": "2026-08-25T11:04:55.594782601Z", "endTime": "2026-08-25T11:23:57.646969908Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI Unexpected error: <fmt.wrapError \| 0xc0011b8160>: failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-5zct5rkc97n4" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00460c52-4267-4ccc-90ce-ef17247c788e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00460c52-4267-4ccc-90ce-ef17247c788e", "name": "00460c52-4267-4ccc-90ce-ef17247c788e", "status": "Failed", "startTime": "2026-08-25T11:04:55.594782601Z", "endTime": "2026-08-25T11:23:57.646969908Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: failed to create HCP cluster np-update-nodes-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000d7cac0>: failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-t54b5mhfsfw7" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/585a4096-9e4c-4106-b5b2-f0f9922f81a9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/585a4096-9e4c-4106-b5b2-f0f9922f81a9", "name": "585a4096-9e4c-4106-b5b2-f0f9922f81a9", "status": "Failed", "startTime": "2026-08-25T11:04:46.951801482Z", "endTime": "2026-08-25T11:23:47.66244911Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: failed to create HCP cluster np-update-nodes-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000d7cac0>: failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-t54b5mhfsfw7" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/585a4096-9e4c-4106-b5b2-f0f9922f81a9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/585a4096-9e4c-4106-b5b2-f0f9922f81a9", "name": "585a4096-9e4c-4106-b5b2-f0f9922f81a9", "status": "Failed", "startTime": "2026-08-25T11:04:46.951801482Z", "endTime": "2026-08-25T11:23:47.66244911Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator, ingress-operator]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc0001908a0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-dcpt8jbnmqxn" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/07a64aff-4896-425c-bd97-6b0eb5c713b8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/07a64aff-4896-425c-bd97-6b0eb5c713b8", "name": "07a64aff-4896-425c-bd97-6b0eb5c713b8", "status": "Failed", "startTime": "2026-08-25T11:06:27.482080484Z", "endTime": "2026-08-25T11:25:27.710374186Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc0001908a0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-dcpt8jbnmqxn" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/07a64aff-4896-425c-bd97-6b0eb5c713b8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/07a64aff-4896-425c-bd97-6b0eb5c713b8", "name": "07a64aff-4896-425c-bd97-6b0eb5c713b8", "status": "Failed", "startTime": "2026-08-25T11:06:27.482080484Z", "endTime": "2026-08-25T11:25:27.710374186Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:98]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0xc0006eefa0>: failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-blffsk8cqz4z" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7cb49989-b0d0-4959-b0b9-b9f58628cdd6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7cb49989-b0d0-4959-b0b9-b9f58628cdd6", "name": "7cb49989-b0d0-4959-b0b9-b9f58628cdd6", "status": "Failed", "startTime": "2026-08-25T11:05:07.01277565Z", "endTime": "2026-08-25T11:24:07.575415707Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:98]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0xc0006eefa0>: failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-blffsk8cqz4z" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7cb49989-b0d0-4959-b0b9-b9f58628cdd6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7cb49989-b0d0-4959-b0b9-b9f58628cdd6", "name": "7cb49989-b0d0-4959-b0b9-b9f58628cdd6", "status": "Failed", "startTime": "2026-08-25T11:05:07.01277565Z", "endTime": "2026-08-25T11:24:07.575415707Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-node-tuning-operator, ignition-server-proxy, openshift-controller-manager, openshift-route-controller-manager]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test Unexpected error: <fmt.wrapError \| 0xc000a56680>: failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-h8x2lnscjddq" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/306db8ae-e454-406c-b2d9-5d6ef9785b20 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/306db8ae-e454-406c-b2d9-5d6ef9785b20", "name": "306db8ae-e454-406c-b2d9-5d6ef9785b20", "status": "Failed", "startTime": "2026-08-25T11:05:01.374469415Z", "endTime": "2026-08-25T11:24:07.731322209Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, openshift-route-controller-manager, cluster-node-tuning-operator; hosted cluster degraded: UnavailableReplicas: [cluster-node-tuning-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 2 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test Unexpected error: <fmt.wrapError \| 0xc000a56680>: failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-h8x2lnscjddq" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/306db8ae-e454-406c-b2d9-5d6ef9785b20 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/306db8ae-e454-406c-b2d9-5d6ef9785b20", "name": "306db8ae-e454-406c-b2d9-5d6ef9785b20", "status": "Failed", "startTime": "2026-08-25T11:05:01.374469415Z", "endTime": "2026-08-25T11:24:07.731322209Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, openshift-route-controller-manager, cluster-node-tuning-operator; hosted cluster degraded: UnavailableReplicas: [cluster-node-tuning-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 2 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test SRE can pause schedules to stop backup execution for an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc001c189e0>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-r5zg79zklshc" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/460dce50-0e02-4f7d-918e-303c53d250cd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/460dce50-0e02-4f7d-918e-303c53d250cd", "name": "460dce50-0e02-4f7d-918e-303c53d250cd", "status": "Failed", "startTime": "2026-08-25T11:05:06.369521114Z", "endTime": "2026-08-25T11:24:07.709199647Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc001c189e0>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-r5zg79zklshc" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/460dce50-0e02-4f7d-918e-303c53d250cd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/460dce50-0e02-4f7d-918e-303c53d250cd", "name": "460dce50-0e02-4f7d-918e-303c53d250cd", "status": "Failed", "startTime": "2026-08-25T11:05:06.369521114Z", "endTime": "2026-08-25T11:24:07.709199647Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred Cluster provisioning failed 10:03:00 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [ingress-operator]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:581]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc0012005e0>: failed to create HCP cluster np-downgrade-sxtp4r: failed waiting for cluster="np-downgrade-sxtp4r" in resourcegroup="rg-np-downgrade-sxtp4r-fhklq2vsgmt4" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34134851-0102-4d35-97b3-944ff4b7f92e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34134851-0102-4d35-97b3-944ff4b7f92e", "name": "34134851-0102-4d35-97b3-944ff4b7f92e", "status": "Failed", "startTime": "2026-08-25T11:04:47.779072949Z", "endTime": "2026-08-25T11:23:47.559079763Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator; hosted cluster degraded: UnavailableReplicas: ingress-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:581]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc0012005e0>: failed to create HCP cluster np-downgrade-sxtp4r: failed waiting for cluster="np-downgrade-sxtp4r" in resourcegroup="rg-np-downgrade-sxtp4r-fhklq2vsgmt4" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34134851-0102-4d35-97b3-944ff4b7f92e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34134851-0102-4d35-97b3-944ff4b7f92e", "name": "34134851-0102-4d35-97b3-944ff4b7f92e", "status": "Failed", "startTime": "2026-08-25T11:04:47.779072949Z", "endTime": "2026-08-25T11:23:47.559079763Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator; hosted cluster degraded: UnavailableReplicas: ingress-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, packageserver]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling Unexpected error: <fmt.wrapError \| 0xc000f38340>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-4zrvgxxs9sbf" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b9cf654f-aed9-4e51-9d7f-39b0aaef1fa4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b9cf654f-aed9-4e51-9d7f-39b0aaef1fa4", "name": "b9cf654f-aed9-4e51-9d7f-39b0aaef1fa4", "status": "Failed", "startTime": "2026-08-25T11:05:16.036728583Z", "endTime": "2026-08-25T11:24:17.88745643Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, openshift-route-controller-manager, redhat-operators-catalog, olm-operator, community-operators-catalog, cluster-policy-controller, cluster-node-tuning-operator, redhat-marketplace-catalog, cluster-storage-operator, packageserver, hosted-cluster-config-operator, ingress-operator, csi-snapshot-controller-operator, openshift-controller-manager, certified-operators-catalog, cluster-version-operator, cluster-network-operator, dns-operator, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling Unexpected error: <fmt.wrapError \| 0xc000f38340>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-4zrvgxxs9sbf" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b9cf654f-aed9-4e51-9d7f-39b0aaef1fa4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b9cf654f-aed9-4e51-9d7f-39b0aaef1fa4", "name": "b9cf654f-aed9-4e51-9d7f-39b0aaef1fa4", "status": "Failed", "startTime": "2026-08-25T11:05:16.036728583Z", "endTime": "2026-08-25T11:24:17.88745643Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, openshift-route-controller-manager, redhat-operators-catalog, olm-operator, community-operators-catalog, cluster-policy-controller, cluster-node-tuning-operator, redhat-marketplace-catalog, cluster-storage-operator, packageserver, hosted-cluster-config-operator, ingress-operator, csi-snapshot-controller-operator, openshift-controller-manager, certified-operators-catalog, cluster-version-operator, cluster-network-operator, dns-operator, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [csi-snapshot-controller-operator, dns-operator, ignition-server-proxy, packageserver]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc001100500>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-29j67455cd7q" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f324379d-3b99-49b4-a8dd-49ba6e3433d2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f324379d-3b99-49b4-a8dd-49ba6e3433d2", "name": "f324379d-3b99-49b4-a8dd-49ba6e3433d2", "status": "Failed", "startTime": "2026-08-25T11:05:06.29234345Z", "endTime": "2026-08-25T11:24:07.6554084Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, dns-operator, cluster-storage-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc001100500>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-29j67455cd7q" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f324379d-3b99-49b4-a8dd-49ba6e3433d2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f324379d-3b99-49b4-a8dd-49ba6e3433d2", "name": "f324379d-3b99-49b4-a8dd-49ba6e3433d2", "status": "Failed", "startTime": "2026-08-25T11:05:06.29234345Z", "endTime": "2026-08-25T11:24:07.6554084Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, dns-operator, cluster-storage-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:147]: failed to create HCP cluster "cp-ystream-upgrade-4-21-cj86x9" at install version 4.20 Unexpected error: <fmt.wrapError \| 0xc000a38640>: failed to create HCP cluster cp-ystream-upgrade-4-21-cj86x9: failed waiting for cluster="cp-ystream-upgrade-4-21-cj86x9" in resourcegroup="rg-cp-ystream-upgrade-4-21-4kqr78rwwtdk" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/df914af8-60c2-4694-8fd1-ddb67d75fb05 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/df914af8-60c2-4694-8fd1-ddb67d75fb05", "name": "df914af8-60c2-4694-8fd1-ddb67d75fb05", "status": "Failed", "startTime": "2026-08-25T11:04:36.064777855Z", "endTime": "2026-08-25T11:23:37.679439509Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:147]: failed to create HCP cluster "cp-ystream-upgrade-4-21-cj86x9" at install version 4.20 Unexpected error: <fmt.wrapError \| 0xc000a38640>: failed to create HCP cluster cp-ystream-upgrade-4-21-cj86x9: failed waiting for cluster="cp-ystream-upgrade-4-21-cj86x9" in resourcegroup="rg-cp-ystream-upgrade-4-21-4kqr78rwwtdk" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/df914af8-60c2-4694-8fd1-ddb67d75fb05 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/df914af8-60c2-4694-8fd1-ddb67d75fb05", "name": "df914af8-60c2-4694-8fd1-ddb67d75fb05", "status": "Failed", "startTime": "2026-08-25T11:04:36.064777855Z", "endTime": "2026-08-25T11:23:37.679439509Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-controller-manager]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-jp97dz" with version 4.21.29 on candidate channel Unexpected error: <fmt.wrapError \| 0xc00105a4a0>: failed to create HCP cluster cluster-zstream-4-21-jp97dz: failed waiting for cluster="cluster-zstream-4-21-jp97dz" in resourcegroup="rg-zstream-upgrade-4-21-9x6gv5vxjxw2" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8798bc57-9343-430d-a747-ca9fa46740a4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8798bc57-9343-430d-a747-ca9fa46740a4", "name": "8798bc57-9343-430d-a747-ca9fa46740a4", "status": "Failed", "startTime": "2026-08-25T11:05:00.721933214Z", "endTime": "2026-08-25T11:24:07.72807329Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 2 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-jp97dz" with version 4.21.29 on candidate channel Unexpected error: <fmt.wrapError \| 0xc00105a4a0>: failed to create HCP cluster cluster-zstream-4-21-jp97dz: failed waiting for cluster="cluster-zstream-4-21-jp97dz" in resourcegroup="rg-zstream-upgrade-4-21-9x6gv5vxjxw2" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8798bc57-9343-430d-a747-ca9fa46740a4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8798bc57-9343-430d-a747-ca9fa46740a4", "name": "8798bc57-9343-430d-a747-ca9fa46740a4", "status": "Failed", "startTime": "2026-08-25T11:05:00.721933214Z", "endTime": "2026-08-25T11:24:07.72807329Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 2 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: WaitingForAvailable: Waiting for Kube APIServer deployment to become available; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-wndrtf" with version 4.20.34 on candidate channel Unexpected error: <fmt.wrapError \| 0xc00132c740>: failed to create HCP cluster cluster-zstream-4-20-wndrtf: failed waiting for cluster="cluster-zstream-4-20-wndrtf" in resourcegroup="rg-zstream-upgrade-4-20-btw2nf2rz2z4" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a719e0dc-6b35-4541-b9c0-4edfc6bb07b8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a719e0dc-6b35-4541-b9c0-4edfc6bb07b8", "name": "a719e0dc-6b35-4541-b9c0-4edfc6bb07b8", "status": "Failed", "startTime": "2026-08-25T11:05:06.275646571Z", "endTime": "2026-08-25T11:24:07.685798394Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: WaitingForAvailable: Waiting for Kube APIServer deployment to become available; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 2 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-wndrtf" with version 4.20.34 on candidate channel Unexpected error: <fmt.wrapError \| 0xc00132c740>: failed to create HCP cluster cluster-zstream-4-20-wndrtf: failed waiting for cluster="cluster-zstream-4-20-wndrtf" in resourcegroup="rg-zstream-upgrade-4-20-btw2nf2rz2z4" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a719e0dc-6b35-4541-b9c0-4edfc6bb07b8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a719e0dc-6b35-4541-b9c0-4edfc6bb07b8", "name": "a719e0dc-6b35-4541-b9c0-4edfc6bb07b8", "status": "Failed", "startTime": "2026-08-25T11:05:06.275646571Z", "endTime": "2026-08-25T11:24:07.685798394Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: WaitingForAvailable: Waiting for Kube APIServer deployment to become available; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 2 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator, ignition-server-proxy, kube-controller-manager, router]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc000d81e60>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-sx5z6bzj9gt4" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9107795d-3348-494d-85dd-e9b48aaf131e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9107795d-3348-494d-85dd-e9b48aaf131e", "name": "9107795d-3348-494d-85dd-e9b48aaf131e", "status": "Failed", "startTime": "2026-08-25T11:05:17.129504958Z", "endTime": "2026-08-25T11:24:17.573823206Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc000d81e60>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-sx5z6bzj9gt4" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9107795d-3348-494d-85dd-e9b48aaf131e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9107795d-3348-494d-85dd-e9b48aaf131e", "name": "9107795d-3348-494d-85dd-e9b48aaf131e", "status": "Failed", "startTime": "2026-08-25T11:05:17.129504958Z", "endTime": "2026-08-25T11:24:17.573823206Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, cluster-network-operator, ingress-operator; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:721]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc001a2ee20>: failed to create HCP cluster np-dg-minor-8wczf9: failed waiting for cluster="np-dg-minor-8wczf9" in resourcegroup="rg-np-dg-minor-8wczf9-z98rv7l55r2s" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/35e222d0-bdde-4e12-a2e6-3658ebb699b6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/35e222d0-bdde-4e12-a2e6-3658ebb699b6", "name": "35e222d0-bdde-4e12-a2e6-3658ebb699b6", "status": "Failed", "startTime": "2026-08-25T11:04:58.632569496Z", "endTime": "2026-08-25T11:24:07.68601483Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, ingress-operator, cluster-image-registry-operator" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:721]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc001a2ee20>: failed to create HCP cluster np-dg-minor-8wczf9: failed waiting for cluster="np-dg-minor-8wczf9" in resourcegroup="rg-np-dg-minor-8wczf9-z98rv7l55r2s" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/35e222d0-bdde-4e12-a2e6-3658ebb699b6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/35e222d0-bdde-4e12-a2e6-3658ebb699b6", "name": "35e222d0-bdde-4e12-a2e6-3658ebb699b6", "status": "Failed", "startTime": "2026-08-25T11:04:58.632569496Z", "endTime": "2026-08-25T11:24:07.68601483Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, ingress-operator, cluster-image-registry-operator" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster Unexpected error: <fmt.wrapError \| 0xc00168cbc0>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-w6bmf26mtnwq" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e950dfd5-2c0f-4378-b9fd-609f947fd461 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e950dfd5-2c0f-4378-b9fd-609f947fd461", "name": "e950dfd5-2c0f-4378-b9fd-609f947fd461", "status": "Failed", "startTime": "2026-08-25T11:04:47.491403607Z", "endTime": "2026-08-25T11:23:47.628860736Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster Unexpected error: <fmt.wrapError \| 0xc00168cbc0>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-w6bmf26mtnwq" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e950dfd5-2c0f-4378-b9fd-609f947fd461 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e950dfd5-2c0f-4378-b9fd-609f947fd461", "name": "e950dfd5-2c0f-4378-b9fd-609f947fd461", "status": "Failed", "startTime": "2026-08-25T11:04:47.491403607Z", "endTime": "2026-08-25T11:23:47.628860736Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [ingress-operator, openshift-controller-manager]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-qpdbsl with version 4.21.30 Unexpected error: <fmt.wrapError \| 0xc000e015c0>: failed to create HCP cluster np-version-upgrade-cluster-qpdbsl: failed waiting for cluster="np-version-upgrade-cluster-qpdbsl" in resourcegroup="rg-np-version-upgrade-qpdbsl-xxx55fsnfv5d" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/35eaac99-6821-4f6d-8ae1-1eb7916b1e9c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/35eaac99-6821-4f6d-8ae1-1eb7916b1e9c", "name": "35eaac99-6821-4f6d-8ae1-1eb7916b1e9c", "status": "Failed", "startTime": "2026-08-25T11:05:07.081081979Z", "endTime": "2026-08-25T11:24:07.516329781Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [ingress-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-qpdbsl with version 4.21.30 Unexpected error: <fmt.wrapError \| 0xc000e015c0>: failed to create HCP cluster np-version-upgrade-cluster-qpdbsl: failed waiting for cluster="np-version-upgrade-cluster-qpdbsl" in resourcegroup="rg-np-version-upgrade-qpdbsl-xxx55fsnfv5d" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/35eaac99-6821-4f6d-8ae1-1eb7916b1e9c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/35eaac99-6821-4f6d-8ae1-1eb7916b1e9c", "name": "35eaac99-6821-4f6d-8ae1-1eb7916b1e9c", "status": "Failed", "startTime": "2026-08-25T11:05:07.081081979Z", "endTime": "2026-08-25T11:24:07.516329781Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [ingress-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, openshift-route-controller-manager, router]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc00058a120>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-hbkfwxhrsvb2" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51ad145b-c45f-488c-9b63-9a804b8af221 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51ad145b-c45f-488c-9b63-9a804b8af221", "name": "51ad145b-c45f-488c-9b63-9a804b8af221", "status": "Failed", "startTime": "2026-08-25T11:05:15.619518812Z", "endTime": "2026-08-25T11:24:17.933561304Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc00058a120>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-hbkfwxhrsvb2" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51ad145b-c45f-488c-9b63-9a804b8af221 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51ad145b-c45f-488c-9b63-9a804b8af221", "name": "51ad145b-c45f-488c-9b63-9a804b8af221", "status": "Failed", "startTime": "2026-08-25T11:05:15.619518812Z", "endTime": "2026-08-25T11:24:17.933561304Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:217]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0xc001580b40>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-t82c95mkn57k" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/36a63320-22f0-4cb1-8ba5-4acf72f853e4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/36a63320-22f0-4cb1-8ba5-4acf72f853e4", "name": "36a63320-22f0-4cb1-8ba5-4acf72f853e4", "status": "Failed", "startTime": "2026-08-25T11:04:56.165986465Z", "endTime": "2026-08-25T11:23:57.660959397Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, cluster-network-operator, dns-operator, ingress-operator, csi-snapshot-controller-operator, cluster-node-tuning-operator, catalog-operator, packageserver, olm-operator, cluster-version-operator, hosted-cluster-config-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:217]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0xc001580b40>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-t82c95mkn57k" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/36a63320-22f0-4cb1-8ba5-4acf72f853e4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/36a63320-22f0-4cb1-8ba5-4acf72f853e4", "name": "36a63320-22f0-4cb1-8ba5-4acf72f853e4", "status": "Failed", "startTime": "2026-08-25T11:04:56.165986465Z", "endTime": "2026-08-25T11:23:57.660959397Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, cluster-network-operator, dns-operator, ingress-operator, csi-snapshot-controller-operator, cluster-node-tuning-operator, catalog-operator, packageserver, olm-operator, cluster-version-operator, hosted-cluster-config-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator, dns-operator, ignition-server-proxy, kube-controller-manager, packageserver]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test SRE should return 409 when boot diagnostics is disabled on a VM · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:417]: failed to create HCP cluster "sre-hcp-cluster-bootdiag" for boot diagnostics test Unexpected error: <fmt.wrapError \| 0xc00059c3e0>: failed to create HCP cluster sre-hcp-cluster-bootdiag: failed waiting for cluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-24c5bjzcljlb" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b21aa181-e22f-4f73-9442-34e06ecf2205 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b21aa181-e22f-4f73-9442-34e06ecf2205", "name": "b21aa181-e22f-4f73-9442-34e06ecf2205", "status": "Failed", "startTime": "2026-08-25T11:04:55.381643689Z", "endTime": "2026-08-25T11:23:57.677492232Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, dns-operator; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 2 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:417]: failed to create HCP cluster "sre-hcp-cluster-bootdiag" for boot diagnostics test Unexpected error: <fmt.wrapError \| 0xc00059c3e0>: failed to create HCP cluster sre-hcp-cluster-bootdiag: failed waiting for cluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-24c5bjzcljlb" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b21aa181-e22f-4f73-9442-34e06ecf2205 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b21aa181-e22f-4f73-9442-34e06ecf2205", "name": "b21aa181-e22f-4f73-9442-34e06ecf2205", "status": "Failed", "startTime": "2026-08-25T11:04:55.381643689Z", "endTime": "2026-08-25T11:23:57.677492232Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, dns-operator; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 2 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--9khvwrw7r7ct/cluster-hs-pre-55dn5b should provision Unexpected error: <fmt.wrapError \| 0xc001046980>: failed to create HCP cluster cluster-hs-pre-55dn5b: failed waiting for cluster="cluster-hs-pre-55dn5b" in resourcegroup="rg-hs-pre--9khvwrw7r7ct" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b00a246e-816f-415f-ad18-11833d6b8316 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b00a246e-816f-415f-ad18-11833d6b8316", "name": "b00a246e-816f-415f-ad18-11833d6b8316", "status": "Failed", "startTime": "2026-08-25T11:05:05.925330051Z", "endTime": "2026-08-25T11:24:07.740218263Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--9khvwrw7r7ct/cluster-hs-pre-55dn5b should provision Unexpected error: <fmt.wrapError \| 0xc001046980>: failed to create HCP cluster cluster-hs-pre-55dn5b: failed waiting for cluster="cluster-hs-pre-55dn5b" in resourcegroup="rg-hs-pre--9khvwrw7r7ct" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b00a246e-816f-415f-ad18-11833d6b8316 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b00a246e-816f-415f-ad18-11833d6b8316", "name": "b00a246e-816f-415f-ad18-11833d6b8316", "status": "Failed", "startTime": "2026-08-25T11:05:05.925330051Z", "endTime": "2026-08-25T11:24:07.740218263Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:147]: failed to create HCP cluster "cp-ystream-upgrade-4-22-wnvltf" at install version 4.21 Unexpected error: <fmt.wrapError \| 0xc0013d4240>: failed to create HCP cluster cp-ystream-upgrade-4-22-wnvltf: failed waiting for cluster="cp-ystream-upgrade-4-22-wnvltf" in resourcegroup="rg-cp-ystream-upgrade-4-22-gt4769dgnqtg" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4573595a-316c-4f3b-b40c-f401e277ab44 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4573595a-316c-4f3b-b40c-f401e277ab44", "name": "4573595a-316c-4f3b-b40c-f401e277ab44", "status": "Failed", "startTime": "2026-08-25T11:05:07.142641521Z", "endTime": "2026-08-25T11:24:07.734693222Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, hosted-cluster-config-operator, olm-operator, certified-operators-catalog, cluster-storage-operator, cluster-node-tuning-operator, dns-operator, community-operators-catalog, cluster-version-operator, cluster-image-registry-operator, packageserver, redhat-marketplace-catalog, openshift-route-controller-manager, openshift-controller-manager, cluster-policy-controller, cluster-network-operator, catalog-operator, redhat-operators-catalog, ingress-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:147]: failed to create HCP cluster "cp-ystream-upgrade-4-22-wnvltf" at install version 4.21 Unexpected error: <fmt.wrapError \| 0xc0013d4240>: failed to create HCP cluster cp-ystream-upgrade-4-22-wnvltf: failed waiting for cluster="cp-ystream-upgrade-4-22-wnvltf" in resourcegroup="rg-cp-ystream-upgrade-4-22-gt4769dgnqtg" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4573595a-316c-4f3b-b40c-f401e277ab44 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4573595a-316c-4f3b-b40c-f401e277ab44", "name": "4573595a-316c-4f3b-b40c-f401e277ab44", "status": "Failed", "startTime": "2026-08-25T11:05:07.142641521Z", "endTime": "2026-08-25T11:24:07.734693222Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, hosted-cluster-config-operator, olm-operator, certified-operators-catalog, cluster-storage-operator, cluster-node-tuning-operator, dns-operator, community-operators-catalog, cluster-version-operator, cluster-image-registry-operator, packageserver, redhat-marketplace-catalog, openshift-route-controller-manager, openshift-controller-manager, cluster-policy-controller, cluster-network-operator, catalog-operator, redhat-operators-catalog, ingress-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260630preview and verify the ingress is internal · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc000f5ae80>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-jpmgv9dccmdw" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0ce80b32-21f8-4a00-a1a0-eb828a65f28d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0ce80b32-21f8-4a00-a1a0-eb828a65f28d", "name": "0ce80b32-21f8-4a00-a1a0-eb828a65f28d", "status": "Failed", "startTime": "2026-08-25T11:04:45.150172423Z", "endTime": "2026-08-25T11:23:47.559518518Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc000f5ae80>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-jpmgv9dccmdw" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0ce80b32-21f8-4a00-a1a0-eb828a65f28d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0ce80b32-21f8-4a00-a1a0-eb828a65f28d", "name": "0ce80b32-21f8-4a00-a1a0-eb828a65f28d", "status": "Failed", "startTime": "2026-08-25T11:04:45.150172423Z", "endTime": "2026-08-25T11:23:47.559518518Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc0009b39a0>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-g89x2rmfntgq" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ed45382-6139-4927-aeb6-b2580f808168 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ed45382-6139-4927-aeb6-b2580f808168", "name": "8ed45382-6139-4927-aeb6-b2580f808168", "status": "Failed", "startTime": "2026-08-25T11:07:09.765576241Z", "endTime": "2026-08-25T11:26:17.763320122Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc0009b39a0>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-g89x2rmfntgq" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ed45382-6139-4927-aeb6-b2580f808168 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ed45382-6139-4927-aeb6-b2580f808168", "name": "8ed45382-6139-4927-aeb6-b2580f808168", "status": "Failed", "startTime": "2026-08-25T11:07:09.765576241Z", "endTime": "2026-08-25T11:26:17.763320122Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:88]: failed to create HCP cluster for kusto logs test Unexpected error: <fmt.wrapError \| 0xc0019a8a40>: failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-x45zf5lld2jq" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f5b946af-4c8b-4b08-8c58-6a22ea330788 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f5b946af-4c8b-4b08-8c58-6a22ea330788", "name": "f5b946af-4c8b-4b08-8c58-6a22ea330788", "status": "Failed", "startTime": "2026-08-25T11:05:07.32799472Z", "endTime": "2026-08-25T11:24:07.638983129Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:88]: failed to create HCP cluster for kusto logs test Unexpected error: <fmt.wrapError \| 0xc0019a8a40>: failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-x45zf5lld2jq" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f5b946af-4c8b-4b08-8c58-6a22ea330788 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f5b946af-4c8b-4b08-8c58-6a22ea330788", "name": "f5b946af-4c8b-4b08-8c58-6a22ea330788", "status": "Failed", "startTime": "2026-08-25T11:05:07.32799472Z", "endTime": "2026-08-25T11:24:07.638983129Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager, catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test SRE should be able to retrieve serial console logs for a VM · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:308]: failed to create HCP cluster "sre-hcp-cluster-sc" for serial console test Unexpected error: <fmt.wrapError \| 0xc000cfa540>: failed to create HCP cluster sre-hcp-cluster-sc: failed waiting for cluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-6b79jbxxk2jv" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51cea064-8c25-47d5-9e4c-98159ead62b6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51cea064-8c25-47d5-9e4c-98159ead62b6", "name": "51cea064-8c25-47d5-9e4c-98159ead62b6", "status": "Failed", "startTime": "2026-08-25T11:04:45.544982989Z", "endTime": "2026-08-25T11:23:47.560554329Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: community-operators-catalog, cluster-version-operator, cluster-network-operator, olm-operator, cluster-storage-operator, redhat-operators-catalog, openshift-controller-manager, openshift-route-controller-manager, ingress-operator, csi-snapshot-controller-operator, redhat-marketplace-catalog, catalog-operator, cluster-policy-controller, cluster-node-tuning-operator, dns-operator, certified-operators-catalog, hosted-cluster-config-operator, azure-cloud-controller-manager, cluster-image-registry-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:308]: failed to create HCP cluster "sre-hcp-cluster-sc" for serial console test Unexpected error: <fmt.wrapError \| 0xc000cfa540>: failed to create HCP cluster sre-hcp-cluster-sc: failed waiting for cluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-6b79jbxxk2jv" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51cea064-8c25-47d5-9e4c-98159ead62b6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51cea064-8c25-47d5-9e4c-98159ead62b6", "name": "51cea064-8c25-47d5-9e4c-98159ead62b6", "status": "Failed", "startTime": "2026-08-25T11:04:45.544982989Z", "endTime": "2026-08-25T11:23:47.560554329Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: community-operators-catalog, cluster-version-operator, cluster-network-operator, olm-operator, cluster-storage-operator, redhat-operators-catalog, openshift-controller-manager, openshift-route-controller-manager, ingress-operator, csi-snapshot-controller-operator, redhat-marketplace-catalog, catalog-operator, cluster-policy-controller, cluster-node-tuning-operator, dns-operator, certified-operators-catalog, hosted-cluster-config-operator, azure-cloud-controller-manager, cluster-image-registry-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, packageserver]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128" Unexpected error: <fmt.wrapError \| 0xc001116340>: failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-bjqmncjbznvt" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/64e0d347-b695-455c-b828-66deb96d7452 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/64e0d347-b695-455c-b828-66deb96d7452", "name": "64e0d347-b695-455c-b828-66deb96d7452", "status": "Failed", "startTime": "2026-08-25T11:05:05.579389204Z", "endTime": "2026-08-25T11:24:07.68094221Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, olm-operator, openshift-route-controller-manager, openshift-controller-manager, cluster-node-tuning-operator, csi-snapshot-controller-operator, catalog-operator, certified-operators-catalog, packageserver, cluster-network-operator, ingress-operator, cluster-version-operator, openshift-apiserver, dns-operator, cluster-policy-controller, redhat-marketplace-catalog, community-operators-catalog, hosted-cluster-config-operator, cluster-storage-operator, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128" Unexpected error: <fmt.wrapError \| 0xc001116340>: failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-bjqmncjbznvt" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/64e0d347-b695-455c-b828-66deb96d7452 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/64e0d347-b695-455c-b828-66deb96d7452", "name": "64e0d347-b695-455c-b828-66deb96d7452", "status": "Failed", "startTime": "2026-08-25T11:05:05.579389204Z", "endTime": "2026-08-25T11:24:07.68094221Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, olm-operator, openshift-route-controller-manager, openshift-controller-manager, cluster-node-tuning-operator, csi-snapshot-controller-operator, catalog-operator, certified-operators-catalog, packageserver, cluster-network-operator, ingress-operator, cluster-version-operator, openshift-apiserver, dns-operator, cluster-policy-controller, redhat-marketplace-catalog, community-operators-catalog, hosted-cluster-config-operator, cluster-storage-operator, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: azure-cloud-controller-manager, catalog-operator, certified-operators-catalog, cluster-autoscaler, cluster-image-registry-operator, cluster-network-operator, cluster-node-tuning-operator, cluster-policy-controller, cluster-storage-operator, cluster-version-operator, community-operators-catalog, control-plane-pki-operator, csi-snapshot-controller-operator, dns-operator, hosted-cluster-config-operator, ignition-server, ignition-server-proxy, ingress-operator, konnectivity-agent, kube-controller-manager, kube-scheduler, machine-approver, olm-collect-profiles, olm-operator, openshift-apiserver, openshift-controller-manager, openshift-route-controller-manager, packageserver, redhat-marketplace-catalog, redhat-operators-catalog; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster Unexpected error: <fmt.wrapError \| 0xc0010fa820>: failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-gp4rdpwm7zhd" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ee129a2-c81c-4d02-bea3-92db5bdcc541 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ee129a2-c81c-4d02-bea3-92db5bdcc541", "name": "5ee129a2-c81c-4d02-bea3-92db5bdcc541", "status": "Failed", "startTime": "2026-08-25T11:04:55.463575249Z", "endTime": "2026-08-25T11:23:57.522671815Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: konnectivity-agent, olm-collect-profiles, cluster-policy-controller, cluster-node-tuning-operator, certified-operators-catalog, cluster-storage-operator, dns-operator, cluster-image-registry-operator, packageserver, hosted-cluster-config-operator, cluster-network-operator, ignition-server-proxy, catalog-operator, openshift-apiserver, ignition-server, olm-operator, redhat-operators-catalog, redhat-marketplace-catalog, community-operators-catalog, machine-approver, kube-controller-manager, openshift-controller-manager, csi-snapshot-controller-operator, control-plane-pki-operator, kube-scheduler, cluster-version-operator, cluster-autoscaler, openshift-route-controller-manager, azure-cloud-controller-manager, ingress-operator" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster Unexpected error: <fmt.wrapError \| 0xc0010fa820>: failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-gp4rdpwm7zhd" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ee129a2-c81c-4d02-bea3-92db5bdcc541 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ee129a2-c81c-4d02-bea3-92db5bdcc541", "name": "5ee129a2-c81c-4d02-bea3-92db5bdcc541", "status": "Failed", "startTime": "2026-08-25T11:04:55.463575249Z", "endTime": "2026-08-25T11:23:57.522671815Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: konnectivity-agent, olm-collect-profiles, cluster-policy-controller, cluster-node-tuning-operator, certified-operators-catalog, cluster-storage-operator, dns-operator, cluster-image-registry-operator, packageserver, hosted-cluster-config-operator, cluster-network-operator, ignition-server-proxy, catalog-operator, openshift-apiserver, ignition-server, olm-operator, redhat-operators-catalog, redhat-marketplace-catalog, community-operators-catalog, machine-approver, kube-controller-manager, openshift-controller-manager, csi-snapshot-controller-operator, control-plane-pki-operator, kube-scheduler, cluster-version-operator, cluster-autoscaler, openshift-route-controller-manager, azure-cloud-controller-manager, ingress-operator" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-node-tuning-operator, openshift-controller-manager; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '20.000000' minutes) Unexpected error: <exported.ResponseError \| 0xc001081c80>: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f7baee5b-db11-4464-aa8e-6e2d27501249 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f7baee5b-db11-4464-aa8e-6e2d27501249", "name": "f7baee5b-db11-4464-aa8e-6e2d27501249", "status": "Failed", "startTime": "2026-08-25T11:06:19.502868721Z", "endTime": "2026-08-25T11:25:27.633996529Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, cluster-node-tuning-operator" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '20.000000' minutes) Unexpected error: <exported.ResponseError \| 0xc001081c80>: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f7baee5b-db11-4464-aa8e-6e2d27501249 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f7baee5b-db11-4464-aa8e-6e2d27501249", "name": "f7baee5b-db11-4464-aa8e-6e2d27501249", "status": "Failed", "startTime": "2026-08-25T11:06:19.502868721Z", "endTime": "2026-08-25T11:25:27.633996529Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, cluster-node-tuning-operator" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc000fb08a0>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-twjmtksnvtqp" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/83902b43-eb0b-4d5c-8306-2571781dc775 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/83902b43-eb0b-4d5c-8306-2571781dc775", "name": "83902b43-eb0b-4d5c-8306-2571781dc775", "status": "Failed", "startTime": "2026-08-25T11:05:07.672178796Z", "endTime": "2026-08-25T11:24:07.708626146Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, catalog-operator, packageserver, cluster-node-tuning-operator, cluster-network-operator, cluster-image-registry-operator, dns-operator, ingress-operator, cluster-storage-operator, olm-operator, hosted-cluster-config-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 2 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc000fb08a0>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-twjmtksnvtqp" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/83902b43-eb0b-4d5c-8306-2571781dc775 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/83902b43-eb0b-4d5c-8306-2571781dc775", "name": "83902b43-eb0b-4d5c-8306-2571781dc775", "status": "Failed", "startTime": "2026-08-25T11:05:07.672178796Z", "endTime": "2026-08-25T11:24:07.708626146Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, catalog-operator, packageserver, cluster-node-tuning-operator, cluster-network-operator, cluster-image-registry-operator, dns-operator, ingress-operator, cluster-storage-operator, olm-operator, hosted-cluster-config-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 2 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator, ignition-server-proxy]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-xk9vbv" with version 5.0.0-ec.5 on candidate channel Unexpected error: <fmt.wrapError \| 0xc001000ca0>: failed to create HCP cluster cluster-zstream-5-0-xk9vbv: failed waiting for cluster="cluster-zstream-5-0-xk9vbv" in resourcegroup="rg-zstream-upgrade-5-0-9zdkrc5gfnqq" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7c9f8e41-e479-4f6e-9ade-6570233d1d04 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7c9f8e41-e479-4f6e-9ade-6570233d1d04", "name": "7c9f8e41-e479-4f6e-9ade-6570233d1d04", "status": "Failed", "startTime": "2026-08-25T11:05:05.893167006Z", "endTime": "2026-08-25T11:24:07.716194982Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, dns-operator; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-xk9vbv" with version 5.0.0-ec.5 on candidate channel Unexpected error: <fmt.wrapError \| 0xc001000ca0>: failed to create HCP cluster cluster-zstream-5-0-xk9vbv: failed waiting for cluster="cluster-zstream-5-0-xk9vbv" in resourcegroup="rg-zstream-upgrade-5-0-9zdkrc5gfnqq" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7c9f8e41-e479-4f6e-9ade-6570233d1d04 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7c9f8e41-e479-4f6e-9ade-6570233d1d04", "name": "7c9f8e41-e479-4f6e-9ade-6570233d1d04", "status": "Failed", "startTime": "2026-08-25T11:05:05.893167006Z", "endTime": "2026-08-25T11:24:07.716194982Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, dns-operator; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager, catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:302]: failed to create HCP cluster np-noedge-wmc2th Unexpected error: <fmt.wrapError \| 0xc000c80de0>: failed to create HCP cluster np-noedge-wmc2th: failed waiting for cluster="np-noedge-wmc2th" in resourcegroup="rg-np-noedge-wmc2th-55l76kt7fdfw" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/599e72ae-426e-413c-a454-0c456b2c46e6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/599e72ae-426e-413c-a454-0c456b2c46e6", "name": "599e72ae-426e-413c-a454-0c456b2c46e6", "status": "Failed", "startTime": "2026-08-25T11:04:46.955311835Z", "endTime": "2026-08-25T11:23:47.543149183Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-apiserver, openshift-controller-manager, hosted-cluster-config-operator, ingress-operator, packageserver, cluster-version-operator, cluster-policy-controller, cluster-storage-operator, dns-operator, olm-operator, cluster-image-registry-operator, redhat-operators-catalog, community-operators-catalog, azure-cloud-controller-manager, cluster-node-tuning-operator, csi-snapshot-controller-operator, certified-operators-catalog, cluster-network-operator, catalog-operator, redhat-marketplace-catalog, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 2 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:302]: failed to create HCP cluster np-noedge-wmc2th Unexpected error: <fmt.wrapError \| 0xc000c80de0>: failed to create HCP cluster np-noedge-wmc2th: failed waiting for cluster="np-noedge-wmc2th" in resourcegroup="rg-np-noedge-wmc2th-55l76kt7fdfw" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/599e72ae-426e-413c-a454-0c456b2c46e6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/599e72ae-426e-413c-a454-0c456b2c46e6", "name": "599e72ae-426e-413c-a454-0c456b2c46e6", "status": "Failed", "startTime": "2026-08-25T11:04:46.955311835Z", "endTime": "2026-08-25T11:23:47.543149183Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-apiserver, openshift-controller-manager, hosted-cluster-config-operator, ingress-operator, packageserver, cluster-version-operator, cluster-policy-controller, cluster-storage-operator, dns-operator, olm-operator, cluster-image-registry-operator, redhat-operators-catalog, community-operators-catalog, azure-cloud-controller-manager, cluster-node-tuning-operator, csi-snapshot-controller-operator, certified-operators-catalog, cluster-network-operator, catalog-operator, redhat-marketplace-catalog, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 2 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, cluster-version-operator, hosted-cluster-config-operator, olm-operator, packageserver]; provider Microsoft.RedHatOpenShift 10:03:00 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc001095660>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-b4qlj57lcxr8" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c3ad12b2-96cb-4af1-a3eb-a6c632d7eb05 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c3ad12b2-96cb-4af1-a3eb-a6c632d7eb05", "name": "c3ad12b2-96cb-4af1-a3eb-a6c632d7eb05", "status": "Failed", "startTime": "2026-08-25T11:05:28.224401746Z", "endTime": "2026-08-25T11:24:37.557730725Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, hosted-cluster-config-operator, cluster-storage-operator, cluster-node-tuning-operator, cluster-network-operator, community-operators-catalog, certified-operators-catalog, cluster-policy-controller, olm-operator, packageserver, redhat-operators-catalog, dns-operator, ingress-operator, cluster-version-operator, openshift-controller-manager, csi-snapshot-controller-operator, cluster-image-registry-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc001095660>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-b4qlj57lcxr8" to finish creating: GET https://rp.j2121984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c3ad12b2-96cb-4af1-a3eb-a6c632d7eb05 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c3ad12b2-96cb-4af1-a3eb-a6c632d7eb05", "name": "c3ad12b2-96cb-4af1-a3eb-a6c632d7eb05", "status": "Failed", "startTime": "2026-08-25T11:05:28.224401746Z", "endTime": "2026-08-25T11:24:37.557730725Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, hosted-cluster-config-operator, cluster-storage-operator, cluster-node-tuning-operator, cluster-network-operator, community-operators-catalog, certified-operators-catalog, cluster-policy-controller, olm-operator, packageserver, redhat-operators-catalog, dns-operator, ingress-operator, cluster-version-operator, openshift-controller-manager, csi-snapshot-controller-operator, cluster-image-registry-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred alert (3) alert [svc] UJClusterProvisionErrors6h30m fired 10:03:00 UTC · failed at alert · test [aro-hcp-observability] [svc] alert UJClusterProvisionErrors6h30m does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-08-25T11:55:05Z Severity: Sev3 Labels: alertname="UJClusterProvisionErrors6h30m", cluster="ci01-j2121984-svc", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2121984/providers/microsoft.monitor/accounts/services-j2121984", severity="3", short_window="30m", slo="cluster-provision-errors" Description: More than 30% of cluster create (install) operations are in failed state sustained over 30 minutes, indicating a medium error budget burn (6x) that would exhaust the 95% SLO budget in ~28 hours. alert [svc] UJClusterProvisionErrors1h5m fired 10:03:00 UTC · failed at alert · test [aro-hcp-observability] [svc] alert UJClusterProvisionErrors1h5m does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-08-25T11:31:06Z Severity: Sev3 Labels: alertname="UJClusterProvisionErrors1h5m", cluster="ci01-j2121984-svc", component="slo", long_window="1h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2121984/providers/microsoft.monitor/accounts/services-j2121984", severity="3", short_window="5m", slo="cluster-provision-errors" Description: More than 72% of cluster create (install) operations are in failed state, indicating a fast error budget burn (14.4x) that would exhaust the 95% SLO budget in ~12 hours. A regional install failure of this magnitude typically points at a shared dependency (e.g. registry, DNS, or ARM) rather than individual clusters. alert [hcp] KubePodNotReady fired 10:03:00 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 214 time(s) (72 unknown, 142 known) Firing 1: State: Resolved Started: 2026-08-25T11:15:53Z Ended: 2026-08-25T11:21:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihoqoqjvco52utltqalrje33t0l73-r7q7g1t2h6y9g9u", pod="etcd-1", severity="warning" Description: Pod ocm-arohcpci01-2sdihoqoqjvco52utltqalrje33t0l73-r7q7g1t2h6y9g9u/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-08-25T11:15:53Z Ended: 2026-08-25T11:22:57Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihornf3ggkrokf3iq4ad00tndcp59-n3t7d0y6p7g0y6a", pod="etcd-0", severity="warning" Description: Pod ocm-arohcpci01-2sdihornf3ggkrokf3iq4ad00tndcp59-n3t7d0y6p7g0y6a/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-08-25T11:15:53Z Ended: 2026-08-25T11:22:57Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihojlg7t1ptd1385e8guhsjjbk24b-m7y2e5b2p6u7y0j", pod="etcd-2", severity="warning" Description: Pod ocm-arohcpci01-2sdihojlg7t1ptd1385e8guhsjjbk24b-m7y2e5b2p6u7y0j/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-08-25T11:15:53Z Ended: 2026-08-25T11:21:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdiiat3uo7ueo5ragnuq92ks0dk69c7-n6g1w0h5p4t6v7x", pod="etcd-0", severity="warning" Description: Pod ocm-arohcpci01-2sdiiat3uo7ueo5ragnuq92ks0dk69c7-n6g1w0h5p4t6v7x/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 5: State: Resolved Started: 2026-08-25T11:15:53Z Ended: 2026-08-25T11:21:59Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihoil30komhjm90bd4i3lf52t0p7e-n4f8u2r9f9j9f6a", pod="etcd-1", severity="warning" Description: Pod ocm-arohcpci01-2sdihoil30komhjm90bd4i3lf52t0p7e-n4f8u2r9f9j9f6a/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 6: State: Resolved Started: 2026-08-25T11:15:53Z Ended: 2026-08-25T11:22:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihm6t4u7847ruif0skle6hdqdtfu0-oidc-wi-cluster", pod="etcd-0", severity="warning" Description: Pod ocm-arohcpci01-2sdihm6t4u7847ruif0skle6hdqdtfu0-oidc-wi-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 7: State: Resolved Started: 2026-08-25T11:15:53Z Ended: 2026-08-25T11:21:59Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdiiat3uo7ueo5ragnuq92ks0dk69c7-n6g1w0h5p4t6v7x", pod="etcd-1", severity="warning" Description: Pod ocm-arohcpci01-2sdiiat3uo7ueo5ragnuq92ks0dk69c7-n6g1w0h5p4t6v7x/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 8: State: Resolved Started: 2026-08-25T11:15:53Z Ended: 2026-08-25T11:22:57Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihm6t4u7847ruif0skle6hdqdtfu0-oidc-wi-cluster", pod="etcd-2", severity="warning" Description: Pod ocm-arohcpci01-2sdihm6t4u7847ruif0skle6hdqdtfu0-oidc-wi-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 9: State: Resolved Started: 2026-08-25T11:16:58Z Ended: 2026-08-25T11:22:57Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihm4a8cjib0rtu5naaoqj6fodpb40-basic-cluster", pod="etcd-2", severity="warning" Description: Pod ocm-arohcpci01-2sdihm4a8cjib0rtu5naaoqj6fodpb40-basic-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 10: State: Resolved Started: 2026-08-25T11:16:58Z Ended: 2026-08-25T11:22:57Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihoqo22gh0a0hddp22l6hd9jkoifi-c3d6z1s0d6g5c0i", pod="router-88b4485c8-6spc7", severity="warning" Description: Pod ocm-arohcpci01-2sdihoqo22gh0a0hddp22l6hd9jkoifi-c3d6z1s0d6g5c0i/router-88b4485c8-6spc7 has been in a non-ready state for longer than 5 minutes. Firing 11: State: Resolved Started: 2026-08-25T11:17:53Z Ended: 2026-08-25T11:26:55Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdiico87tbt5b77mctlj7ovpf2bj5rv-cilium-cluster", pod="router-65fcc9bc6d-7cmzc", severity="warning" Description: Pod ocm-arohcpci01-2sdiico87tbt5b77mctlj7ovpf2bj5rv-cilium-cluster/router-65fcc9bc6d-7cmzc has been in a non-ready state for longer than 5 minutes. Firing 12: State: Resolved Started: 2026-08-25T11:17:53Z Ended: 2026-08-25T11:23:57Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihrhpvfule14t15r9pd9emls5ui4e-b3l7a0h4c9y3h5i", pod="router-8475d5578f-kw7rk", severity="warning" Description: Pod ocm-arohcpci01-2sdihrhpvfule14t15r9pd9emls5ui4e-b3l7a0h4c9y3h5i/router-8475d5578f-kw7rk has been in a non-ready state for longer than 5 minutes. Firing 13: State: Resolved Started: 2026-08-25T11:17:53Z Ended: 2026-08-25T11:32:54Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihp0q2ddeb5fgluh885f3vnifv11v-q8l9x8o4f8k6i4u", pod="router-77b6469bc4-kdxjp", severity="warning" Description: Pod ocm-arohcpci01-2sdihp0q2ddeb5fgluh885f3vnifv11v-q8l9x8o4f8k6i4u/router-77b6469bc4-kdxjp has been in a non-ready state for longer than 5 minutes. Firing 14: State: Resolved Started: 2026-08-25T11:18:57Z Ended: 2026-08-25T11:24:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihk0jjm46dcuek7fkdv1s0mlma270-b8y5y3n9s1i3e6r", pod="kube-apiserver-684bdb74f4-qrhlb", severity="warning" Description: Pod ocm-arohcpci01-2sdihk0jjm46dcuek7fkdv1s0mlma270-b8y5y3n9s1i3e6r/kube-apiserver-684bdb74f4-qrhlb has been in a non-ready state for longer than 5 minutes. Firing 15: State: Resolved Started: 2026-08-25T11:19:55Z Ended: 2026-08-25T11:25:55Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihjm1mhlr4psccldumle6ua3dk7h1-l8c7m3s0e0m1c0f", pod="kube-apiserver-56bc7995d6-qk57b", severity="warning" Description: Pod ocm-arohcpci01-2sdihjm1mhlr4psccldumle6ua3dk7h1-l8c7m3s0e0m1c0f/kube-apiserver-56bc7995d6-qk57b has been in a non-ready state for longer than 5 minutes. Firing 16: State: Resolved Started: 2026-08-25T11:19:55Z Ended: 2026-08-25T11:25:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihlpvvvkmgro6e6csbku88ncb9en6-s0x4p8y9x0c6v4m", pod="kube-apiserver-b6f44b495-zr7g8", severity="warning" Description: Pod ocm-arohcpci01-2sdihlpvvvkmgro6e6csbku88ncb9en6-s0x4p8y9x0c6v4m/kube-apiserver-b6f44b495-zr7g8 has been in a non-ready state for longer than 5 minutes. Firing 17: State: Resolved Started: 2026-08-25T11:19:55Z Ended: 2026-08-25T11:25:55Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdiiavncdrhqeuttup2n9hg48qes0ha-g4d1n8t0u3e4d7d", pod="cluster-node-tuning-operator-56ff6789cc-nkt22", severity="warning" Description: Pod ocm-arohcpci01-2sdiiavncdrhqeuttup2n9hg48qes0ha-g4d1n8t0u3e4d7d/cluster-node-tuning-operator-56ff6789cc-nkt22 has been in a non-ready state for longer than 5 minutes. Firing 18: State: Resolved Started: 2026-08-25T11:19:55Z Ended: 2026-08-25T11:25:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihk0f5j57g03nmcd0mogmv2dt2nku-z8h9z4y3f3u5t3o", pod="openshift-apiserver-dff87b4bd-g5j9z", severity="warning" Description: Pod ocm-arohcpci01-2sdihk0f5j57g03nmcd0mogmv2dt2nku-z8h9z4y3f3u5t3o/openshift-apiserver-dff87b4bd-g5j9z has been in a non-ready state for longer than 5 minutes. Firing 19: State: Resolved Started: 2026-08-25T11:19:55Z Ended: 2026-08-25T11:25:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihonffubcm8u116hplvg0t738r0a1-j9l0h3i4s3p1o5k", pod="kube-apiserver-6745758fb9-j5pnz", severity="warning" Description: Pod ocm-arohcpci01-2sdihonffubcm8u116hplvg0t738r0a1-j9l0h3i4s3p1o5k/kube-apiserver-6745758fb9-j5pnz has been in a non-ready state for longer than 5 minutes. Firing 20: State: Resolved Started: 2026-08-25T11:19:55Z Ended: 2026-08-25T11:25:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihn8kefnu9ef3p1g5eoidqlfh9uch-c5i6v3u9t2p2s5b", pod="openshift-apiserver-b9cdd8b74-llmwt", severity="warning" Description: Pod ocm-arohcpci01-2sdihn8kefnu9ef3p1g5eoidqlfh9uch-c5i6v3u9t2p2s5b/openshift-apiserver-b9cdd8b74-llmwt has been in a non-ready state for longer than 5 minutes. Firing 21: State: Resolved Started: 2026-08-25T11:19:55Z Ended: 2026-08-25T11:25:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihooatb2qifmu342ldo11dqubp742-b5f2i5i5a5b3l2r", pod="kube-apiserver-7c8b6d766f-hmccd", severity="warning" Description: Pod ocm-arohcpci01-2sdihooatb2qifmu342ldo11dqubp742-b5f2i5i5a5b3l2r/kube-apiserver-7c8b6d766f-hmccd has been in a non-ready state for longer than 5 minutes. Firing 22: State: Resolved Started: 2026-08-25T11:19:55Z Ended: 2026-08-25T11:25:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdiiavncdrhqeuttup2n9hg48qes0ha-g4d1n8t0u3e4d7d", pod="openshift-controller-manager-7646ffc7c8-988nr", severity="warning" Description: Pod ocm-arohcpci01-2sdiiavncdrhqeuttup2n9hg48qes0ha-g4d1n8t0u3e4d7d/openshift-controller-manager-7646ffc7c8-988nr has been in a non-ready state for longer than 5 minutes. Firing 23: State: Resolved Started: 2026-08-25T11:20:53Z Ended: 2026-08-25T11:27:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihp5qru6coctlgq10q529qq8q8t3k-s0z6a1q4s4a5y3k", pod="cluster-storage-operator-78ffcb4d4c-lbg55", severity="warning" Description: Pod ocm-arohcpci01-2sdihp5qru6coctlgq10q529qq8q8t3k-s0z6a1q4s4a5y3k/cluster-storage-operator-78ffcb4d4c-lbg55 has been in a non-ready state for longer than 5 minutes. Firing 24: State: Resolved Started: 2026-08-25T11:20:53Z Ended: 2026-08-25T11:27:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihp5qru6coctlgq10q529qq8q8t3k-s0z6a1q4s4a5y3k", pod="dns-operator-f98478b88-dx44g", severity="warning" Description: Pod ocm-arohcpci01-2sdihp5qru6coctlgq10q529qq8q8t3k-s0z6a1q4s4a5y3k/dns-operator-f98478b88-dx44g has been in a non-ready state for longer than 5 minutes. Firing 25: State: Resolved Started: 2026-08-25T11:21:57Z Ended: 2026-08-25T11:27:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihm0uplesi9d5agopa3d0t0oe12qi-cilium-cl", pod="ingress-operator-846bbf64f7-v6nwx", severity="warning" Description: Pod ocm-arohcpci01-2sdihm0uplesi9d5agopa3d0t0oe12qi-cilium-cl/ingress-operator-846bbf64f7-v6nwx has been in a non-ready state for longer than 5 minutes. Firing 26: State: Resolved Started: 2026-08-25T11:21:57Z Ended: 2026-08-25T11:27:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihm0uplesi9d5agopa3d0t0oe12qi-cilium-cl", pod="cluster-network-operator-b897ddf64-4v29h", severity="warning" Description: Pod ocm-arohcpci01-2sdihm0uplesi9d5agopa3d0t0oe12qi-cilium-cl/cluster-network-operator-b897ddf64-4v29h has been in a non-ready state for longer than 5 minutes. Firing 27: State: Resolved Started: 2026-08-25T11:21:57Z Ended: 2026-08-25T11:27:55Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihm0uplesi9d5agopa3d0t0oe12qi-cilium-cl", pod="dns-operator-858b5f4f74-zzxv2", severity="warning" Description: Pod ocm-arohcpci01-2sdihm0uplesi9d5agopa3d0t0oe12qi-cilium-cl/dns-operator-858b5f4f74-zzxv2 has been in a non-ready state for longer than 5 minutes. Firing 28: State: Resolved Started: 2026-08-25T11:21:57Z Ended: 2026-08-25T11:27:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihud59ft0p329osbaf1otr9ejbg1p-private-kas", pod="kube-apiserver-5f969fff47-mvbml", severity="warning" Description: Pod ocm-arohcpci01-2sdihud59ft0p329osbaf1otr9ejbg1p-private-kas/kube-apiserver-5f969fff47-mvbml has been in a non-ready state for longer than 5 minutes. Firing 29: State: Resolved Started: 2026-08-25T11:21:57Z Ended: 2026-08-25T11:27:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihjqfseqk0bkameel5g0dcmelgj1p-ea-cluster", pod="dns-operator-576db85c76-7hgp4", severity="warning" Description: Pod ocm-arohcpci01-2sdihjqfseqk0bkameel5g0dcmelgj1p-ea-cluster/dns-operator-576db85c76-7hgp4 has been in a non-ready state for longer than 5 minutes. Firing 30: State: Resolved Started: 2026-08-25T11:21:57Z Ended: 2026-08-25T11:27:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihk2iajl1nss25pddg85qcnt48735-u9g4x2e7k1c4y9t", pod="azure-cloud-controller-manager-7cb5476c8c-t78kj", severity="warning" Description: Pod ocm-arohcpci01-2sdihk2iajl1nss25pddg85qcnt48735-u9g4x2e7k1c4y9t/azure-cloud-controller-manager-7cb5476c8c-t78kj has been in a non-ready state for longer than 5 minutes. Firing 31: State: Resolved Started: 2026-08-25T11:21:57Z Ended: 2026-08-25T11:27:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihp5qru6coctlgq10q529qq8q8t3k-s0z6a1q4s4a5y3k", pod="dns-operator-5bc9f885d9-ngvtl", severity="warning" Description: Pod ocm-arohcpci01-2sdihp5qru6coctlgq10q529qq8q8t3k-s0z6a1q4s4a5y3k/dns-operator-5bc9f885d9-ngvtl has been in a non-ready state for longer than 5 minutes. Firing 32: State: Resolved Started: 2026-08-25T11:21:57Z Ended: 2026-08-25T11:27:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihr2jf10qmgvg3b8sudro0sel69i3-t0x2s4i2j0p9y5k", pod="cluster-node-tuning-operator-7c97fc476c-jrzh2", severity="warning" Description: Pod ocm-arohcpci01-2sdihr2jf10qmgvg3b8sudro0sel69i3-t0x2s4i2j0p9y5k/cluster-node-tuning-operator-7c97fc476c-jrzh2 has been in a non-ready state for longer than 5 minutes. Firing 33: State: Resolved Started: 2026-08-25T11:21:57Z Ended: 2026-08-25T11:33:55Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihr3d1lisej0e8km2sm69rbb5btmv-n1k8p5v8e7s2o2y", pod="router-7647c6448d-mb5j7", severity="warning" Description: Pod ocm-arohcpci01-2sdihr3d1lisej0e8km2sm69rbb5btmv-n1k8p5v8e7s2o2y/router-7647c6448d-mb5j7 has been in a non-ready state for longer than 5 minutes. Firing 34: State: Resolved Started: 2026-08-25T11:21:57Z Ended: 2026-08-25T11:27:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihm0uplesi9d5agopa3d0t0oe12qi-cilium-cl", pod="csi-snapshot-controller-operator-7857d4b9b-dljzv", severity="warning" Description: Pod ocm-arohcpci01-2sdihm0uplesi9d5agopa3d0t0oe12qi-cilium-cl/csi-snapshot-controller-operator-7857d4b9b-dljzv has been in a non-ready state for longer than 5 minutes. Firing 35: State: Resolved Started: 2026-08-25T11:21:57Z Ended: 2026-08-25T11:27:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihnlofblh7399cq4fbi0l5tqtvvn4-e1w9t6c4k1o3x0u", pod="azure-cloud-controller-manager-9f68d94cb-n7fvw", severity="warning" Description: Pod ocm-arohcpci01-2sdihnlofblh7399cq4fbi0l5tqtvvn4-e1w9t6c4k1o3x0u/azure-cloud-controller-manager-9f68d94cb-n7fvw has been in a non-ready state for longer than 5 minutes. Firing 36: State: Resolved Started: 2026-08-25T11:22:56Z Ended: 2026-08-25T11:30:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihogf52emihtpnapdqhgjjda1t302-g0u4p3s3b9m3g5r", pod="cluster-image-registry-operator-86d46445b-62xt9", severity="warning" Description: Pod ocm-arohcpci01-2sdihogf52emihtpnapdqhgjjda1t302-g0u4p3s3b9m3g5r/cluster-image-registry-operator-86d46445b-62xt9 has been in a non-ready state for longer than 5 minutes. Firing 37: State: Resolved Started: 2026-08-25T11:22:56Z Ended: 2026-08-25T11:33:55Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihoqo22gh0a0hddp22l6hd9jkoifi-c3d6z1s0d6g5c0i", pod="router-785ddc79dd-dvmfd", severity="warning" Description: Pod ocm-arohcpci01-2sdihoqo22gh0a0hddp22l6hd9jkoifi-c3d6z1s0d6g5c0i/router-785ddc79dd-dvmfd has been in a non-ready state for longer than 5 minutes. Firing 38: State: Resolved Started: 2026-08-25T11:22:56Z Ended: 2026-08-25T11:29:55Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihov0r88snbbbcn0aeubuu3akj96i-ephemeral-disk", pod="cluster-storage-operator-79b759d5b6-6bbz6", severity="warning" Description: Pod ocm-arohcpci01-2sdihov0r88snbbbcn0aeubuu3akj96i-ephemeral-disk/cluster-storage-operator-79b759d5b6-6bbz6 has been in a non-ready state for longer than 5 minutes. Firing 39: State: Resolved Started: 2026-08-25T11:22:56Z Ended: 2026-08-25T11:29:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihojlg7t1ptd1385e8guhsjjbk24b-m7y2e5b2p6u7y0j", pod="kube-apiserver-c48b9b6f7-jgrct", severity="warning" Description: Pod ocm-arohcpci01-2sdihojlg7t1ptd1385e8guhsjjbk24b-m7y2e5b2p6u7y0j/kube-apiserver-c48b9b6f7-jgrct has been in a non-ready state for longer than 5 minutes. Firing 40: State: Resolved Started: 2026-08-25T11:22:56Z Ended: 2026-08-25T11:29:57Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihm7cgmbogpmampkrspaq5hon0cos-y1d8u1t5b8e3q7j", pod="kube-apiserver-b987c96f4-cfhp8", severity="warning" Description: Pod ocm-arohcpci01-2sdihm7cgmbogpmampkrspaq5hon0cos-y1d8u1t5b8e3q7j/kube-apiserver-b987c96f4-cfhp8 has been in a non-ready state for longer than 5 minutes. Firing 41: State: Resolved Started: 2026-08-25T11:22:57Z Ended: 2026-08-25T11:29:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihm7cgmbogpmampkrspaq5hon0cos-y1d8u1t5b8e3q7j", pod="kube-apiserver-b987c96f4-x8snx", severity="warning" Description: Pod ocm-arohcpci01-2sdihm7cgmbogpmampkrspaq5hon0cos-y1d8u1t5b8e3q7j/kube-apiserver-b987c96f4-x8snx has been in a non-ready state for longer than 5 minutes. Firing 42: State: Resolved Started: 2026-08-25T11:23:55Z Ended: 2026-08-25T11:29:59Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihjmdk026fo2kbma08qfa1cejc49u-private-ingress", pod="openshift-apiserver-68f6b7c98b-fzttk", severity="warning" Description: Pod ocm-arohcpci01-2sdihjmdk026fo2kbma08qfa1cejc49u-private-ingress/openshift-apiserver-68f6b7c98b-fzttk has been in a non-ready state for longer than 5 minutes. Firing 43: State: Resolved Started: 2026-08-25T11:23:55Z Ended: 2026-08-25T11:29:55Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdiibbfs9l2infhclpl8df2iq6lbgu9-k1x8g2z9k7d2n2n", pod="openshift-apiserver-cf6d9d745-jsdt4", severity="warning" Description: Pod ocm-arohcpci01-2sdiibbfs9l2infhclpl8df2iq6lbgu9-k1x8g2z9k7d2n2n/openshift-apiserver-cf6d9d745-jsdt4 has been in a non-ready state for longer than 5 minutes. Firing 44: State: Resolved Started: 2026-08-25T11:23:55Z Ended: 2026-08-25T11:29:55Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihp5sbk2173baqaesnoqdga6u4via-v20260901", pod="cluster-storage-operator-776855f9bb-z84fc", severity="warning" Description: Pod ocm-arohcpci01-2sdihp5sbk2173baqaesnoqdga6u4via-v20260901/cluster-storage-operator-776855f9bb-z84fc has been in a non-ready state for longer than 5 minutes. Firing 45: State: Resolved Started: 2026-08-25T11:23:55Z Ended: 2026-08-25T11:29:57Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihjmdk026fo2kbma08qfa1cejc49u-private-ingress", pod="machine-approver-587dbd6f84-mgcg2", severity="warning" Description: Pod ocm-arohcpci01-2sdihjmdk026fo2kbma08qfa1cejc49u-private-ingress/machine-approver-587dbd6f84-mgcg2 has been in a non-ready state for longer than 5 minutes. Firing 46: State: Resolved Started: 2026-08-25T11:23:55Z Ended: 2026-08-25T11:33:54Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihrhpvfule14t15r9pd9emls5ui4e-b3l7a0h4c9y3h5i", pod="router-57fff74698-2zzgx", severity="warning" Description: Pod ocm-arohcpci01-2sdihrhpvfule14t15r9pd9emls5ui4e-b3l7a0h4c9y3h5i/router-57fff74698-2zzgx has been in a non-ready state for longer than 5 minutes. Firing 47: State: Resolved Started: 2026-08-25T11:23:55Z Ended: 2026-08-25T11:29:57Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihp5sbk2173baqaesnoqdga6u4via-v20260901", pod="csi-snapshot-controller-operator-6ccc765f9c-wqphn", severity="warning" Description: Pod ocm-arohcpci01-2sdihp5sbk2173baqaesnoqdga6u4via-v20260901/csi-snapshot-controller-operator-6ccc765f9c-wqphn has been in a non-ready state for longer than 5 minutes. Firing 48: State: Resolved Started: 2026-08-25T11:23:55Z Ended: 2026-08-25T11:29:54Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihp5sbk2173baqaesnoqdga6u4via-v20260901", pod="dns-operator-6666fbcc7-jpqbl", severity="warning" Description: Pod ocm-arohcpci01-2sdihp5sbk2173baqaesnoqdga6u4via-v20260901/dns-operator-6666fbcc7-jpqbl has been in a non-ready state for longer than 5 minutes. Firing 49: State: Resolved Started: 2026-08-25T11:23:55Z Ended: 2026-08-25T11:32:54Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihk2iajl1nss25pddg85qcnt48735-u9g4x2e7k1c4y9t", pod="ingress-operator-64d7b6d557-9njjm", severity="warning" Description: Pod ocm-arohcpci01-2sdihk2iajl1nss25pddg85qcnt48735-u9g4x2e7k1c4y9t/ingress-operator-64d7b6d557-9njjm has been in a non-ready state for longer than 5 minutes. Firing 50: State: Resolved Started: 2026-08-25T11:23:55Z Ended: 2026-08-25T11:29:59Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihoqo22gh0a0hddp22l6hd9jkoifi-c3d6z1s0d6g5c0i", pod="dns-operator-648d47796d-2dghs", severity="warning" Description: Pod ocm-arohcpci01-2sdihoqo22gh0a0hddp22l6hd9jkoifi-c3d6z1s0d6g5c0i/dns-operator-648d47796d-2dghs has been in a non-ready state for longer than 5 minutes. Firing 51: State: Resolved Started: 2026-08-25T11:24:54Z Ended: 2026-08-25T11:30:57Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihoo74po3sqe3cajn5hji6egsflgt-sre-hcp-cluster", pod="openshift-controller-manager-89cf84d4f-zx7s6", severity="warning" Description: Pod ocm-arohcpci01-2sdihoo74po3sqe3cajn5hji6egsflgt-sre-hcp-cluster/openshift-controller-manager-89cf84d4f-zx7s6 has been in a non-ready state for longer than 5 minutes. Firing 52: State: Resolved Started: 2026-08-25T11:24:54Z Ended: 2026-08-25T11:30:55Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihnlofblh7399cq4fbi0l5tqtvvn4-e1w9t6c4k1o3x0u", pod="cluster-node-tuning-operator-c799c98fc-j7mg4", severity="warning" Description: Pod ocm-arohcpci01-2sdihnlofblh7399cq4fbi0l5tqtvvn4-e1w9t6c4k1o3x0u/cluster-node-tuning-operator-c799c98fc-j7mg4 has been in a non-ready state for longer than 5 minutes. Firing 53: State: Resolved Started: 2026-08-25T11:24:54Z Ended: 2026-08-25T11:31:53Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihp0q2ddeb5fgluh885f3vnifv11v-q8l9x8o4f8k6i4u", pod="dns-operator-c596dc85c-r7kmb", severity="warning" Description: Pod ocm-arohcpci01-2sdihp0q2ddeb5fgluh885f3vnifv11v-q8l9x8o4f8k6i4u/dns-operator-c596dc85c-r7kmb has been in a non-ready state for longer than 5 minutes. Firing 54: State: Resolved Started: 2026-08-25T11:24:54Z Ended: 2026-08-25T11:30:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihnlofblh7399cq4fbi0l5tqtvvn4-e1w9t6c4k1o3x0u", pod="openshift-route-controller-manager-864c948c74-ncqjn", severity="warning" Description: Pod ocm-arohcpci01-2sdihnlofblh7399cq4fbi0l5tqtvvn4-e1w9t6c4k1o3x0u/openshift-route-controller-manager-864c948c74-ncqjn has been in a non-ready state for longer than 5 minutes. Firing 55: State: Resolved Started: 2026-08-25T11:24:54Z Ended: 2026-08-25T11:30:57Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdiiat3uo7ueo5ragnuq92ks0dk69c7-n6g1w0h5p4t6v7x", pod="cluster-node-tuning-operator-bbb458659-r9hdr", severity="warning" Description: Pod ocm-arohcpci01-2sdiiat3uo7ueo5ragnuq92ks0dk69c7-n6g1w0h5p4t6v7x/cluster-node-tuning-operator-bbb458659-r9hdr has been in a non-ready state for longer than 5 minutes. Firing 56: State: Resolved Started: 2026-08-25T11:24:54Z Ended: 2026-08-25T11:31:53Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihp0q2ddeb5fgluh885f3vnifv11v-q8l9x8o4f8k6i4u", pod="ingress-operator-5bd9df95c6-l59w6", severity="warning" Description: Pod ocm-arohcpci01-2sdihp0q2ddeb5fgluh885f3vnifv11v-q8l9x8o4f8k6i4u/ingress-operator-5bd9df95c6-l59w6 has been in a non-ready state for longer than 5 minutes. Firing 57: State: Resolved Started: 2026-08-25T11:24:54Z Ended: 2026-08-25T11:30:55Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihnlofblh7399cq4fbi0l5tqtvvn4-e1w9t6c4k1o3x0u", pod="openshift-route-controller-manager-864c948c74-54xlv", severity="warning" Description: Pod ocm-arohcpci01-2sdihnlofblh7399cq4fbi0l5tqtvvn4-e1w9t6c4k1o3x0u/openshift-route-controller-manager-864c948c74-54xlv has been in a non-ready state for longer than 5 minutes. Firing 58: State: Resolved Started: 2026-08-25T11:24:54Z Ended: 2026-08-25T11:30:57Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihm7bk5doidrj40qnllvjids6cmu3-q3l8j5u5k3u1a0k", pod="cluster-network-operator-756fb5c44f-pg7cs", severity="warning" Description: Pod ocm-arohcpci01-2sdihm7bk5doidrj40qnllvjids6cmu3-q3l8j5u5k3u1a0k/cluster-network-operator-756fb5c44f-pg7cs has been in a non-ready state for longer than 5 minutes. Firing 59: State: Resolved Started: 2026-08-25T11:24:54Z Ended: 2026-08-25T11:30:57Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihp42qokd8iirnheso0hhb5iadhlg-a2x2i0t6e1p5b5y", pod="openshift-controller-manager-f44f79c46-kgcnh", severity="warning" Description: Pod ocm-arohcpci01-2sdihp42qokd8iirnheso0hhb5iadhlg-a2x2i0t6e1p5b5y/openshift-controller-manager-f44f79c46-kgcnh has been in a non-ready state for longer than 5 minutes. Firing 60: State: Resolved Started: 2026-08-25T11:24:54Z Ended: 2026-08-25T11:33:54Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihoo74po3sqe3cajn5hji6egsflgt-sre-hcp-cluster", pod="openshift-route-controller-manager-6b4b5b6f99-4jmj5", severity="warning" Description: Pod ocm-arohcpci01-2sdihoo74po3sqe3cajn5hji6egsflgt-sre-hcp-cluster/openshift-route-controller-manager-6b4b5b6f99-4jmj5 has been in a non-ready state for longer than 5 minutes. Firing 61: State: Resolved Started: 2026-08-25T11:24:55Z Ended: 2026-08-25T11:30:57Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihr3d1lisej0e8km2sm69rbb5btmv-n1k8p5v8e7s2o2y", pod="openshift-route-controller-manager-5d79b9bdf6-hddjw", severity="warning" Description: Pod ocm-arohcpci01-2sdihr3d1lisej0e8km2sm69rbb5btmv-n1k8p5v8e7s2o2y/openshift-route-controller-manager-5d79b9bdf6-hddjw has been in a non-ready state for longer than 5 minutes. Firing 62: State: Resolved Started: 2026-08-25T11:24:55Z Ended: 2026-08-25T11:30:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihp0q2ddeb5fgluh885f3vnifv11v-q8l9x8o4f8k6i4u", pod="cluster-node-tuning-operator-756b5dff7b-h48nf", severity="warning" Description: Pod ocm-arohcpci01-2sdihp0q2ddeb5fgluh885f3vnifv11v-q8l9x8o4f8k6i4u/cluster-node-tuning-operator-756b5dff7b-h48nf has been in a non-ready state for longer than 5 minutes. Firing 63: State: Resolved Started: 2026-08-25T11:24:55Z Ended: 2026-08-25T11:30:57Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihr3d1lisej0e8km2sm69rbb5btmv-n1k8p5v8e7s2o2y", pod="cluster-policy-controller-854567cccb-cws4c", severity="warning" Description: Pod ocm-arohcpci01-2sdihr3d1lisej0e8km2sm69rbb5btmv-n1k8p5v8e7s2o2y/cluster-policy-controller-854567cccb-cws4c has been in a non-ready state for longer than 5 minutes. Firing 64: State: Resolved Started: 2026-08-25T11:24:55Z Ended: 2026-08-25T11:30:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihn2lfkgh3lt1mq76md59k1m8bn2f-p9u7t7f4z4a6g8h", pod="openshift-controller-manager-6b6546f4f7-qxkjm", severity="warning" Description: Pod ocm-arohcpci01-2sdihn2lfkgh3lt1mq76md59k1m8bn2f-p9u7t7f4z4a6g8h/openshift-controller-manager-6b6546f4f7-qxkjm has been in a non-ready state for longer than 5 minutes. Firing 65: State: Resolved Started: 2026-08-25T11:24:55Z Ended: 2026-08-25T11:30:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihm7bk5doidrj40qnllvjids6cmu3-q3l8j5u5k3u1a0k", pod="openshift-controller-manager-5db4c96ff7-lscxk", severity="warning" Description: Pod ocm-arohcpci01-2sdihm7bk5doidrj40qnllvjids6cmu3-q3l8j5u5k3u1a0k/openshift-controller-manager-5db4c96ff7-lscxk has been in a non-ready state for longer than 5 minutes. Firing 66: State: Resolved Started: 2026-08-25T11:24:55Z Ended: 2026-08-25T11:30:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihogqel47t0pk7ir633okvtdrk0a2-h7m7s0l5k5s9d3r", pod="cluster-node-tuning-operator-858564c7-v59hc", severity="warning" Description: Pod ocm-arohcpci01-2sdihogqel47t0pk7ir633okvtdrk0a2-h7m7s0l5k5s9d3r/cluster-node-tuning-operator-858564c7-v59hc has been in a non-ready state for longer than 5 minutes. Firing 67: State: Resolved Started: 2026-08-25T11:24:55Z Ended: 2026-08-25T11:30:57Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihnlofblh7399cq4fbi0l5tqtvvn4-e1w9t6c4k1o3x0u", pod="openshift-controller-manager-688457c87b-dr92t", severity="warning" Description: Pod ocm-arohcpci01-2sdihnlofblh7399cq4fbi0l5tqtvvn4-e1w9t6c4k1o3x0u/openshift-controller-manager-688457c87b-dr92t has been in a non-ready state for longer than 5 minutes. Firing 68: State: Resolved Started: 2026-08-25T11:24:55Z Ended: 2026-08-25T11:30:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdiiat3uo7ueo5ragnuq92ks0dk69c7-n6g1w0h5p4t6v7x", pod="openshift-controller-manager-5dcdfbfc5c-7wksr", severity="warning" Description: Pod ocm-arohcpci01-2sdiiat3uo7ueo5ragnuq92ks0dk69c7-n6g1w0h5p4t6v7x/openshift-controller-manager-5dcdfbfc5c-7wksr has been in a non-ready state for longer than 5 minutes. Firing 69: State: Resolved Started: 2026-08-25T11:24:55Z Ended: 2026-08-25T11:30:55Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihp0q2ddeb5fgluh885f3vnifv11v-q8l9x8o4f8k6i4u", pod="cluster-network-operator-698b868fdb-w6q94", severity="warning" Description: Pod ocm-arohcpci01-2sdihp0q2ddeb5fgluh885f3vnifv11v-q8l9x8o4f8k6i4u/cluster-network-operator-698b868fdb-w6q94 has been in a non-ready state for longer than 5 minutes. Firing 70: State: Resolved Started: 2026-08-25T11:24:55Z Ended: 2026-08-25T11:30:55Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihm7bk5doidrj40qnllvjids6cmu3-q3l8j5u5k3u1a0k", pod="ingress-operator-9747d4f68-vb8bl", severity="warning" Description: Pod ocm-arohcpci01-2sdihm7bk5doidrj40qnllvjids6cmu3-q3l8j5u5k3u1a0k/ingress-operator-9747d4f68-vb8bl has been in a non-ready state for longer than 5 minutes. Firing 71: State: Resolved Started: 2026-08-25T11:24:55Z Ended: 2026-08-25T11:30:57Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihog7uq0k1ofaet0tpch6hfnpp1m6-j2z2w3q5v7n7q1v", pod="openshift-apiserver-6cb576bd79-gqdlj", severity="warning" Description: Pod ocm-arohcpci01-2sdihog7uq0k1ofaet0tpch6hfnpp1m6-j2z2w3q5v7n7q1v/openshift-apiserver-6cb576bd79-gqdlj has been in a non-ready state for longer than 5 minutes. Firing 72: State: Resolved Started: 2026-08-25T11:24:55Z Ended: 2026-08-25T11:30:55Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2121984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihp0q2ddeb5fgluh885f3vnifv11v-q8l9x8o4f8k6i4u", pod="cluster-image-registry-operator-548bb6b86c-dss79", severity="warning" Description: Pod ocm-arohcpci01-2sdihp0q2ddeb5fgluh885f3vnifv11v-q8l9x8o4f8k6i4u/cluster-image-registry-operator-548bb6b86c-dss79 has been in a non-ready state for longer than 5 minutes.
10:02:57	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 3c81c74	alert, e2e	failed	⚠#6645 (open)	40	e2e (38) ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints Unexpected error: <fmt.wrapError \| 0xc000f9e500>: failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-kxgdkssgfc6d" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8f655cf-cd5c-4298-b821-2efa60072edd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8f655cf-cd5c-4298-b821-2efa60072edd", "name": "c8f655cf-cd5c-4298-b821-2efa60072edd", "status": "Failed", "startTime": "2026-08-25T11:18:37.705265753Z", "endTime": "2026-08-25T11:37:45.81706187Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints Unexpected error: <fmt.wrapError \| 0xc000f9e500>: failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-kxgdkssgfc6d" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8f655cf-cd5c-4298-b821-2efa60072edd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8f655cf-cd5c-4298-b821-2efa60072edd", "name": "c8f655cf-cd5c-4298-b821-2efa60072edd", "status": "Failed", "startTime": "2026-08-25T11:18:37.705265753Z", "endTime": "2026-08-25T11:37:45.81706187Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator, cluster-network-operator, cluster-storage-operator, csi-snapshot-controller-operator, dns-operator, ingress-operator, packageserver]; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc000e0b0e0>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-57kmhshxtcx2" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b802d9c9-7b78-4adb-ba82-1320188f9c6d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b802d9c9-7b78-4adb-ba82-1320188f9c6d", "name": "b802d9c9-7b78-4adb-ba82-1320188f9c6d", "status": "Failed", "startTime": "2026-08-25T11:02:19.928553979Z", "endTime": "2026-08-25T11:21:25.9465048Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, catalog-operator, cluster-storage-operator, dns-operator, ingress-operator, csi-snapshot-controller-operator, hosted-cluster-config-operator, cluster-image-registry-operator, packageserver, olm-operator; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, packageserver deployment has 2 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc000e0b0e0>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-57kmhshxtcx2" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b802d9c9-7b78-4adb-ba82-1320188f9c6d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b802d9c9-7b78-4adb-ba82-1320188f9c6d", "name": "b802d9c9-7b78-4adb-ba82-1320188f9c6d", "status": "Failed", "startTime": "2026-08-25T11:02:19.928553979Z", "endTime": "2026-08-25T11:21:25.9465048Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, catalog-operator, cluster-storage-operator, dns-operator, ingress-operator, csi-snapshot-controller-operator, hosted-cluster-config-operator, cluster-image-registry-operator, packageserver, olm-operator; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, packageserver deployment has 2 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0xc000f761c0>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-tsn8fjg7z8v8" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7c60a9b4-98e8-437a-b0d1-340131fafe1e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7c60a9b4-98e8-437a-b0d1-340131fafe1e", "name": "7c60a9b4-98e8-437a-b0d1-340131fafe1e", "status": "Failed", "startTime": "2026-08-25T11:02:30.548851412Z", "endTime": "2026-08-25T11:21:35.946063021Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, catalog-operator, cluster-version-operator, openshift-route-controller-manager, olm-operator, packageserver, certified-operators-catalog, cluster-node-tuning-operator, dns-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, redhat-marketplace-catalog, cluster-storage-operator, ingress-operator, redhat-operators-catalog, openshift-controller-manager, community-operators-catalog, hosted-cluster-config-operator, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0xc000f761c0>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-tsn8fjg7z8v8" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7c60a9b4-98e8-437a-b0d1-340131fafe1e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7c60a9b4-98e8-437a-b0d1-340131fafe1e", "name": "7c60a9b4-98e8-437a-b0d1-340131fafe1e", "status": "Failed", "startTime": "2026-08-25T11:02:30.548851412Z", "endTime": "2026-08-25T11:21:35.946063021Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, catalog-operator, cluster-version-operator, openshift-route-controller-manager, olm-operator, packageserver, certified-operators-catalog, cluster-node-tuning-operator, dns-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, redhat-marketplace-catalog, cluster-storage-operator, ingress-operator, redhat-operators-catalog, openshift-controller-manager, community-operators-catalog, hosted-cluster-config-operator, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager, catalog-operator, hosted-cluster-config-operator, machine-approver, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc000ce01c0>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-vm8pwb95swzd" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/17b6acf0-f165-4ca6-b8c3-e2f36eb085c4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/17b6acf0-f165-4ca6-b8c3-e2f36eb085c4", "name": "17b6acf0-f165-4ca6-b8c3-e2f36eb085c4", "status": "Failed", "startTime": "2026-08-25T11:03:11.32352815Z", "endTime": "2026-08-25T11:22:15.832343372Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-node-tuning-operator, cluster-image-registry-operator, dns-operator, kube-storage-version-migrator, cluster-policy-controller, hosted-cluster-config-operator, cluster-storage-operator, catalog-operator, community-operators-catalog, machine-approver, openshift-route-controller-manager, cluster-network-operator, csi-snapshot-controller-operator, packageserver, certified-operators-catalog, openshift-controller-manager, cluster-version-operator, redhat-operators-catalog, openshift-apiserver, ingress-operator, olm-operator, azure-cloud-controller-manager; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 3 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc000ce01c0>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-vm8pwb95swzd" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/17b6acf0-f165-4ca6-b8c3-e2f36eb085c4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/17b6acf0-f165-4ca6-b8c3-e2f36eb085c4", "name": "17b6acf0-f165-4ca6-b8c3-e2f36eb085c4", "status": "Failed", "startTime": "2026-08-25T11:03:11.32352815Z", "endTime": "2026-08-25T11:22:15.832343372Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-node-tuning-operator, cluster-image-registry-operator, dns-operator, kube-storage-version-migrator, cluster-policy-controller, hosted-cluster-config-operator, cluster-storage-operator, catalog-operator, community-operators-catalog, machine-approver, openshift-route-controller-manager, cluster-network-operator, csi-snapshot-controller-operator, packageserver, certified-operators-catalog, openshift-controller-manager, cluster-version-operator, redhat-operators-catalog, openshift-apiserver, ingress-operator, olm-operator, azure-cloud-controller-manager; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 3 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc000e81d20>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-wtmf95jcsb5k" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/14e367e4-ea49-470a-a267-1aa8a5819f26 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/14e367e4-ea49-470a-a267-1aa8a5819f26", "name": "14e367e4-ea49-470a-a267-1aa8a5819f26", "status": "Failed", "startTime": "2026-08-25T11:04:32.694001765Z", "endTime": "2026-08-25T11:23:35.873645251Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, cluster-version-operator, cluster-network-operator, redhat-marketplace-catalog, community-operators-catalog, ingress-operator, openshift-route-controller-manager, cluster-policy-controller, cluster-storage-operator, redhat-operators-catalog, csi-snapshot-controller-operator, catalog-operator, certified-operators-catalog, openshift-controller-manager, dns-operator, cluster-node-tuning-operator, olm-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 3 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 3 unavailable replicas, openshift-route-controller-manager deployment has 3 unavailable replicas, packageserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc000e81d20>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-wtmf95jcsb5k" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/14e367e4-ea49-470a-a267-1aa8a5819f26 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/14e367e4-ea49-470a-a267-1aa8a5819f26", "name": "14e367e4-ea49-470a-a267-1aa8a5819f26", "status": "Failed", "startTime": "2026-08-25T11:04:32.694001765Z", "endTime": "2026-08-25T11:23:35.873645251Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, cluster-version-operator, cluster-network-operator, redhat-marketplace-catalog, community-operators-catalog, ingress-operator, openshift-route-controller-manager, cluster-policy-controller, cluster-storage-operator, redhat-operators-catalog, csi-snapshot-controller-operator, catalog-operator, certified-operators-catalog, openshift-controller-manager, dns-operator, cluster-node-tuning-operator, olm-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 3 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 3 unavailable replicas, openshift-route-controller-manager deployment has 3 unavailable replicas, packageserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster Unexpected error: <fmt.wrapError \| 0xc00021f7c0>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-8s5smmwtzc77" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b32196f2-1fcd-4826-ae2c-d7a59f640453 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b32196f2-1fcd-4826-ae2c-d7a59f640453", "name": "b32196f2-1fcd-4826-ae2c-d7a59f640453", "status": "Failed", "startTime": "2026-08-25T11:02:30.79442888Z", "endTime": "2026-08-25T11:21:35.920091579Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster Unexpected error: <fmt.wrapError \| 0xc00021f7c0>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-8s5smmwtzc77" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b32196f2-1fcd-4826-ae2c-d7a59f640453 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b32196f2-1fcd-4826-ae2c-d7a59f640453", "name": "b32196f2-1fcd-4826-ae2c-d7a59f640453", "status": "Failed", "startTime": "2026-08-25T11:02:30.79442888Z", "endTime": "2026-08-25T11:21:35.920091579Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:98]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0xc000880340>: failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-fncq4c5d429d" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7409d60a-0ba9-4daa-99e6-fd3d757217a6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7409d60a-0ba9-4daa-99e6-fd3d757217a6", "name": "7409d60a-0ba9-4daa-99e6-fd3d757217a6", "status": "Failed", "startTime": "2026-08-25T11:04:21.864024356Z", "endTime": "2026-08-25T11:23:25.850925072Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator, catalog-operator, cluster-network-operator, olm-operator, cluster-version-operator, cluster-storage-operator, dns-operator, csi-snapshot-controller-operator, packageserver, hosted-cluster-config-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 2 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:98]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0xc000880340>: failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-fncq4c5d429d" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7409d60a-0ba9-4daa-99e6-fd3d757217a6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7409d60a-0ba9-4daa-99e6-fd3d757217a6", "name": "7409d60a-0ba9-4daa-99e6-fd3d757217a6", "status": "Failed", "startTime": "2026-08-25T11:04:21.864024356Z", "endTime": "2026-08-25T11:23:25.850925072Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator, catalog-operator, cluster-network-operator, olm-operator, cluster-version-operator, cluster-storage-operator, dns-operator, csi-snapshot-controller-operator, packageserver, hosted-cluster-config-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 2 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, cluster-version-operator]; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:721]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000e7a340>: failed to create HCP cluster np-dg-minor-5k4t7m: failed waiting for cluster="np-dg-minor-5k4t7m" in resourcegroup="rg-np-dg-minor-5k4t7m-c7rmk8jp25j5" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/36c411c9-a482-476f-8c6d-e111f54a5519 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/36c411c9-a482-476f-8c6d-e111f54a5519", "name": "36c411c9-a482-476f-8c6d-e111f54a5519", "status": "Failed", "startTime": "2026-08-25T11:02:30.5276884Z", "endTime": "2026-08-25T11:21:35.939284741Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:721]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000e7a340>: failed to create HCP cluster np-dg-minor-5k4t7m: failed waiting for cluster="np-dg-minor-5k4t7m" in resourcegroup="rg-np-dg-minor-5k4t7m-c7rmk8jp25j5" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/36c411c9-a482-476f-8c6d-e111f54a5519 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/36c411c9-a482-476f-8c6d-e111f54a5519", "name": "36c411c9-a482-476f-8c6d-e111f54a5519", "status": "Failed", "startTime": "2026-08-25T11:02:30.5276884Z", "endTime": "2026-08-25T11:21:35.939284741Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test SRE should be able to retrieve serial console logs for a VM · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:308]: failed to create HCP cluster "sre-hcp-cluster-sc" for serial console test Unexpected error: <fmt.wrapError \| 0xc0018a0760>: failed to create HCP cluster sre-hcp-cluster-sc: failed waiting for cluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-k2ffghkn9d59" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9cdfa7a3-a70a-44de-9fde-fd3953477c72 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9cdfa7a3-a70a-44de-9fde-fd3953477c72", "name": "9cdfa7a3-a70a-44de-9fde-fd3953477c72", "status": "Failed", "startTime": "2026-08-25T11:02:38.97665824Z", "endTime": "2026-08-25T11:21:45.876490122Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:308]: failed to create HCP cluster "sre-hcp-cluster-sc" for serial console test Unexpected error: <fmt.wrapError \| 0xc0018a0760>: failed to create HCP cluster sre-hcp-cluster-sc: failed waiting for cluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-k2ffghkn9d59" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9cdfa7a3-a70a-44de-9fde-fd3953477c72 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9cdfa7a3-a70a-44de-9fde-fd3953477c72", "name": "9cdfa7a3-a70a-44de-9fde-fd3953477c72", "status": "Failed", "startTime": "2026-08-25T11:02:38.97665824Z", "endTime": "2026-08-25T11:21:45.876490122Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-wgqxxx" with version 5.0.0-ec.5 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000f36680>: failed to create HCP cluster cluster-zstream-5-0-wgqxxx: failed waiting for cluster="cluster-zstream-5-0-wgqxxx" in resourcegroup="rg-zstream-upgrade-5-0-dtrxh9srjttj" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2e03a2cc-036c-400d-8fc0-f3fe60b7e8f8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2e03a2cc-036c-400d-8fc0-f3fe60b7e8f8", "name": "2e03a2cc-036c-400d-8fc0-f3fe60b7e8f8", "status": "Failed", "startTime": "2026-08-25T11:02:05.144488342Z", "endTime": "2026-08-25T11:21:05.810048041Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-wgqxxx" with version 5.0.0-ec.5 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000f36680>: failed to create HCP cluster cluster-zstream-5-0-wgqxxx: failed waiting for cluster="cluster-zstream-5-0-wgqxxx" in resourcegroup="rg-zstream-upgrade-5-0-dtrxh9srjttj" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2e03a2cc-036c-400d-8fc0-f3fe60b7e8f8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2e03a2cc-036c-400d-8fc0-f3fe60b7e8f8", "name": "2e03a2cc-036c-400d-8fc0-f3fe60b7e8f8", "status": "Failed", "startTime": "2026-08-25T11:02:05.144488342Z", "endTime": "2026-08-25T11:21:05.810048041Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator, dns-operator, ingress-operator]; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-c2f59xczgxh8/cluster-candidate-4-21-57drd2 should provision Unexpected error: <fmt.wrapError \| 0xc001bba960>: failed to create HCP cluster cluster-candidate-4-21-57drd2: failed waiting for cluster="cluster-candidate-4-21-57drd2" in resourcegroup="rg-candidate-4-21-c2f59xczgxh8" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d8baa3d8-e441-4aa4-ad34-789f804408c7 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d8baa3d8-e441-4aa4-ad34-789f804408c7", "name": "d8baa3d8-e441-4aa4-ad34-789f804408c7", "status": "Failed", "startTime": "2026-08-25T11:02:18.423236357Z", "endTime": "2026-08-25T11:21:25.878162329Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator, cluster-network-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-c2f59xczgxh8/cluster-candidate-4-21-57drd2 should provision Unexpected error: <fmt.wrapError \| 0xc001bba960>: failed to create HCP cluster cluster-candidate-4-21-57drd2: failed waiting for cluster="cluster-candidate-4-21-57drd2" in resourcegroup="rg-candidate-4-21-c2f59xczgxh8" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d8baa3d8-e441-4aa4-ad34-789f804408c7 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d8baa3d8-e441-4aa4-ad34-789f804408c7", "name": "d8baa3d8-e441-4aa4-ad34-789f804408c7", "status": "Failed", "startTime": "2026-08-25T11:02:18.423236357Z", "endTime": "2026-08-25T11:21:25.878162329Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator, cluster-network-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:147]: failed to create HCP cluster "cp-ystream-upgrade-4-22-d4q7n8" at install version 4.21 Unexpected error: <fmt.wrapError \| 0xc00123e240>: failed to create HCP cluster cp-ystream-upgrade-4-22-d4q7n8: failed waiting for cluster="cp-ystream-upgrade-4-22-d4q7n8" in resourcegroup="rg-cp-ystream-upgrade-4-22-skn4bzscn96c" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f32701d7-6200-468b-980b-66f6b8257b49 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f32701d7-6200-468b-980b-66f6b8257b49", "name": "f32701d7-6200-468b-980b-66f6b8257b49", "status": "Failed", "startTime": "2026-08-25T11:02:29.254710411Z", "endTime": "2026-08-25T11:21:35.961594722Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:147]: failed to create HCP cluster "cp-ystream-upgrade-4-22-d4q7n8" at install version 4.21 Unexpected error: <fmt.wrapError \| 0xc00123e240>: failed to create HCP cluster cp-ystream-upgrade-4-22-d4q7n8: failed waiting for cluster="cp-ystream-upgrade-4-22-d4q7n8" in resourcegroup="rg-cp-ystream-upgrade-4-22-skn4bzscn96c" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f32701d7-6200-468b-980b-66f6b8257b49 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f32701d7-6200-468b-980b-66f6b8257b49", "name": "f32701d7-6200-468b-980b-66f6b8257b49", "status": "Failed", "startTime": "2026-08-25T11:02:29.254710411Z", "endTime": "2026-08-25T11:21:35.961594722Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator]; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-zgp8947jk9nr/cluster-candidate-4-22-8d2tgf should provision Unexpected error: <fmt.wrapError \| 0xc000e64200>: failed to create HCP cluster cluster-candidate-4-22-8d2tgf: failed waiting for cluster="cluster-candidate-4-22-8d2tgf" in resourcegroup="rg-candidate-4-22-zgp8947jk9nr" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8fe159a5-0698-4edb-aeeb-bac8baa0530e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8fe159a5-0698-4edb-aeeb-bac8baa0530e", "name": "8fe159a5-0698-4edb-aeeb-bac8baa0530e", "status": "Failed", "startTime": "2026-08-25T11:02:29.272254875Z", "endTime": "2026-08-25T11:21:35.930400552Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-zgp8947jk9nr/cluster-candidate-4-22-8d2tgf should provision Unexpected error: <fmt.wrapError \| 0xc000e64200>: failed to create HCP cluster cluster-candidate-4-22-8d2tgf: failed waiting for cluster="cluster-candidate-4-22-8d2tgf" in resourcegroup="rg-candidate-4-22-zgp8947jk9nr" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8fe159a5-0698-4edb-aeeb-bac8baa0530e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8fe159a5-0698-4edb-aeeb-bac8baa0530e", "name": "8fe159a5-0698-4edb-aeeb-bac8baa0530e", "status": "Failed", "startTime": "2026-08-25T11:02:29.272254875Z", "endTime": "2026-08-25T11:21:35.930400552Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-controller-manager]; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc000f4a700>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-4kwl7rkmbbsz" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/55b4601f-fd16-4b14-83df-ce42375746d5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/55b4601f-fd16-4b14-83df-ce42375746d5", "name": "55b4601f-fd16-4b14-83df-ce42375746d5", "status": "Failed", "startTime": "2026-08-25T11:02:28.796667754Z", "endTime": "2026-08-25T11:21:35.927411954Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 2 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc000f4a700>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-4kwl7rkmbbsz" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/55b4601f-fd16-4b14-83df-ce42375746d5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/55b4601f-fd16-4b14-83df-ce42375746d5", "name": "55b4601f-fd16-4b14-83df-ce42375746d5", "status": "Failed", "startTime": "2026-08-25T11:02:28.796667754Z", "endTime": "2026-08-25T11:21:35.927411954Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 2 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:599]: failed to create nodepool npdg-4-21 Unexpected error: <fmt.wrapError \| 0xc000622d60>: failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-77gllg" in resourcegroup="rg-np-downgrade-77gllg-4r7rgqsrgh66" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e1ce5887-74ea-4460-be42-e4173a17bfbe -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e1ce5887-74ea-4460-be42-e4173a17bfbe", "name": "e1ce5887-74ea-4460-be42-e4173a17bfbe", "status": "Failed", "startTime": "2026-08-25T11:15:36.167245148Z", "endTime": "2026-08-25T11:34:45.86895076Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:599]: failed to create nodepool npdg-4-21 Unexpected error: <fmt.wrapError \| 0xc000622d60>: failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-77gllg" in resourcegroup="rg-np-downgrade-77gllg-4r7rgqsrgh66" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e1ce5887-74ea-4460-be42-e4173a17bfbe -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e1ce5887-74ea-4460-be42-e4173a17bfbe", "name": "e1ce5887-74ea-4460-be42-e4173a17bfbe", "status": "Failed", "startTime": "2026-08-25T11:15:36.167245148Z", "endTime": "2026-08-25T11:34:45.86895076Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:442]: failed to create HCP cluster np-skip-minor-f8plrr at version 4.22.11 Unexpected error: <fmt.wrapError \| 0xc0011dc620>: failed to create HCP cluster np-skip-minor-f8plrr: failed waiting for cluster="np-skip-minor-f8plrr" in resourcegroup="rg-np-skip-minor-f8plrr-kgf8vn4q6tps" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6bf4fc95-3caf-453e-9414-14bf5c581205 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6bf4fc95-3caf-453e-9414-14bf5c581205", "name": "6bf4fc95-3caf-453e-9414-14bf5c581205", "status": "Failed", "startTime": "2026-08-25T11:02:30.549945224Z", "endTime": "2026-08-25T11:21:35.854676439Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, ingress-operator, cluster-image-registry-operator, cluster-policy-controller, cluster-storage-operator, cluster-network-operator, dns-operator, csi-snapshot-controller-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 2 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, packageserver deployment has 2 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:442]: failed to create HCP cluster np-skip-minor-f8plrr at version 4.22.11 Unexpected error: <fmt.wrapError \| 0xc0011dc620>: failed to create HCP cluster np-skip-minor-f8plrr: failed waiting for cluster="np-skip-minor-f8plrr" in resourcegroup="rg-np-skip-minor-f8plrr-kgf8vn4q6tps" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6bf4fc95-3caf-453e-9414-14bf5c581205 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6bf4fc95-3caf-453e-9414-14bf5c581205", "name": "6bf4fc95-3caf-453e-9414-14bf5c581205", "status": "Failed", "startTime": "2026-08-25T11:02:30.549945224Z", "endTime": "2026-08-25T11:21:35.854676439Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, ingress-operator, cluster-image-registry-operator, cluster-policy-controller, cluster-storage-operator, cluster-network-operator, dns-operator, csi-snapshot-controller-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 2 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, packageserver deployment has 2 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster Unexpected error: <fmt.wrapError \| 0xc000ff61c0>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-d6scg6nrwppk" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7bb3d540-0c63-47be-ae4b-406cef03fed4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7bb3d540-0c63-47be-ae4b-406cef03fed4", "name": "7bb3d540-0c63-47be-ae4b-406cef03fed4", "status": "Failed", "startTime": "2026-08-25T11:02:30.420307394Z", "endTime": "2026-08-25T11:21:36.057718306Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster Unexpected error: <fmt.wrapError \| 0xc000ff61c0>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-d6scg6nrwppk" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7bb3d540-0c63-47be-ae4b-406cef03fed4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7bb3d540-0c63-47be-ae4b-406cef03fed4", "name": "7bb3d540-0c63-47be-ae4b-406cef03fed4", "status": "Failed", "startTime": "2026-08-25T11:02:30.420307394Z", "endTime": "2026-08-25T11:21:36.057718306Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test SRE can pause schedules to stop backup execution for an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc001002180>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-qxgjhv2nzd4n" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/359c8bcf-a5fe-4f3a-b5e8-9eeaeaf92731 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/359c8bcf-a5fe-4f3a-b5e8-9eeaeaf92731", "name": "359c8bcf-a5fe-4f3a-b5e8-9eeaeaf92731", "status": "Failed", "startTime": "2026-08-25T11:02:29.8394171Z", "endTime": "2026-08-25T11:21:36.044301402Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-network-operator, cluster-node-tuning-operator, dns-operator, olm-operator, openshift-controller-manager, hosted-cluster-config-operator, cluster-storage-operator, openshift-route-controller-manager, packageserver, csi-snapshot-controller-operator, cluster-version-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 2 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 2 unavailable replicas, openshift-route-controller-manager deployment has 2 unavailable replicas, packageserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc001002180>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-qxgjhv2nzd4n" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/359c8bcf-a5fe-4f3a-b5e8-9eeaeaf92731 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/359c8bcf-a5fe-4f3a-b5e8-9eeaeaf92731", "name": "359c8bcf-a5fe-4f3a-b5e8-9eeaeaf92731", "status": "Failed", "startTime": "2026-08-25T11:02:29.8394171Z", "endTime": "2026-08-25T11:21:36.044301402Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-network-operator, cluster-node-tuning-operator, dns-operator, olm-operator, openshift-controller-manager, hosted-cluster-config-operator, cluster-storage-operator, openshift-route-controller-manager, packageserver, csi-snapshot-controller-operator, cluster-version-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 2 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 2 unavailable replicas, openshift-route-controller-manager deployment has 2 unavailable replicas, packageserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: WaitingForAvailable: Waiting for Kube APIServer deployment to become available; hosted cluster degraded: UnavailableReplicas: [capi-provider, kube-apiserver, router]; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc000f3e1a0>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-nlgtzd8jj8ls" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5c6c05ac-d3c3-4c18-b01d-729aef0c992e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5c6c05ac-d3c3-4c18-b01d-729aef0c992e", "name": "5c6c05ac-d3c3-4c18-b01d-729aef0c992e", "status": "Failed", "startTime": "2026-08-25T11:02:29.862453094Z", "endTime": "2026-08-25T11:21:35.935998302Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: WaitingForAvailable: Waiting for Kube APIServer deployment to become available; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc000f3e1a0>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-nlgtzd8jj8ls" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5c6c05ac-d3c3-4c18-b01d-729aef0c992e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5c6c05ac-d3c3-4c18-b01d-729aef0c992e", "name": "5c6c05ac-d3c3-4c18-b01d-729aef0c992e", "status": "Failed", "startTime": "2026-08-25T11:02:29.862453094Z", "endTime": "2026-08-25T11:21:35.935998302Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: WaitingForAvailable: Waiting for Kube APIServer deployment to become available; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator, dns-operator, ingress-operator]; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-k9tccs with version 4.21.30 Unexpected error: <fmt.wrapError \| 0xc0005c61e0>: failed to create HCP cluster np-version-upgrade-cluster-k9tccs: failed waiting for cluster="np-version-upgrade-cluster-k9tccs" in resourcegroup="rg-np-version-upgrade-k9tccs-dxkthfx2xcmq" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/06b6d562-cc76-49e7-90e9-e0c893ebdaca -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/06b6d562-cc76-49e7-90e9-e0c893ebdaca", "name": "06b6d562-cc76-49e7-90e9-e0c893ebdaca", "status": "Failed", "startTime": "2026-08-25T11:02:30.260205158Z", "endTime": "2026-08-25T11:21:35.932791726Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, dns-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-k9tccs with version 4.21.30 Unexpected error: <fmt.wrapError \| 0xc0005c61e0>: failed to create HCP cluster np-version-upgrade-cluster-k9tccs: failed waiting for cluster="np-version-upgrade-cluster-k9tccs" in resourcegroup="rg-np-version-upgrade-k9tccs-dxkthfx2xcmq" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/06b6d562-cc76-49e7-90e9-e0c893ebdaca -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/06b6d562-cc76-49e7-90e9-e0c893ebdaca", "name": "06b6d562-cc76-49e7-90e9-e0c893ebdaca", "status": "Failed", "startTime": "2026-08-25T11:02:30.260205158Z", "endTime": "2026-08-25T11:21:35.932791726Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, dns-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000d7e7a0>: failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-t47tntcsxrpm" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/afe963d2-3618-4199-8c62-b39caa628b25 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/afe963d2-3618-4199-8c62-b39caa628b25", "name": "afe963d2-3618-4199-8c62-b39caa628b25", "status": "Failed", "startTime": "2026-08-25T11:02:28.270419413Z", "endTime": "2026-08-25T11:21:36.047784787Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, cluster-version-operator, ingress-operator, hosted-cluster-config-operator, dns-operator, olm-operator, cluster-network-operator, catalog-operator, packageserver, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 2 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000d7e7a0>: failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-t47tntcsxrpm" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/afe963d2-3618-4199-8c62-b39caa628b25 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/afe963d2-3618-4199-8c62-b39caa628b25", "name": "afe963d2-3618-4199-8c62-b39caa628b25", "status": "Failed", "startTime": "2026-08-25T11:02:28.270419413Z", "endTime": "2026-08-25T11:21:36.047784787Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, cluster-version-operator, ingress-operator, hosted-cluster-config-operator, dns-operator, olm-operator, cluster-network-operator, catalog-operator, packageserver, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 2 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [capi-provider, kube-apiserver]; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:302]: failed to create HCP cluster np-noedge-vk6cfm Unexpected error: <fmt.wrapError \| 0xc001216240>: failed to create HCP cluster np-noedge-vk6cfm: failed waiting for cluster="np-noedge-vk6cfm" in resourcegroup="rg-np-noedge-vk6cfm-r5z9ggrs4bzc" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6d73055c-caa7-4581-b662-b31a5154ad44 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6d73055c-caa7-4581-b662-b31a5154ad44", "name": "6d73055c-caa7-4581-b662-b31a5154ad44", "status": "Failed", "startTime": "2026-08-25T11:02:40.855489688Z", "endTime": "2026-08-25T11:21:45.894005118Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator, packageserver, olm-collect-profiles, community-operators-catalog, azure-cloud-controller-manager, cluster-storage-operator, konnectivity-agent, cluster-autoscaler, cluster-version-operator, csi-snapshot-controller-operator, catalog-operator, openshift-apiserver, cluster-image-registry-operator, certified-operators-catalog, capi-provider, openshift-route-controller-manager, cluster-policy-controller, hosted-cluster-config-operator, dns-operator, ignition-server, olm-operator, machine-approver, kube-controller-manager, kube-scheduler, cluster-network-operator, control-plane-pki-operator, openshift-controller-manager, redhat-operators-catalog, cluster-node-tuning-operator, ignition-server-proxy, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:302]: failed to create HCP cluster np-noedge-vk6cfm Unexpected error: <fmt.wrapError \| 0xc001216240>: failed to create HCP cluster np-noedge-vk6cfm: failed waiting for cluster="np-noedge-vk6cfm" in resourcegroup="rg-np-noedge-vk6cfm-r5z9ggrs4bzc" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6d73055c-caa7-4581-b662-b31a5154ad44 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6d73055c-caa7-4581-b662-b31a5154ad44", "name": "6d73055c-caa7-4581-b662-b31a5154ad44", "status": "Failed", "startTime": "2026-08-25T11:02:40.855489688Z", "endTime": "2026-08-25T11:21:45.894005118Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator, packageserver, olm-collect-profiles, community-operators-catalog, azure-cloud-controller-manager, cluster-storage-operator, konnectivity-agent, cluster-autoscaler, cluster-version-operator, csi-snapshot-controller-operator, catalog-operator, openshift-apiserver, cluster-image-registry-operator, certified-operators-catalog, capi-provider, openshift-route-controller-manager, cluster-policy-controller, hosted-cluster-config-operator, dns-operator, ignition-server, olm-operator, machine-approver, kube-controller-manager, kube-scheduler, cluster-network-operator, control-plane-pki-operator, openshift-controller-manager, redhat-operators-catalog, cluster-node-tuning-operator, ignition-server-proxy, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260630preview and verify the ingress is internal · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc000d66520>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-k4pms9xqtc7k" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d5ee1a7-b048-4225-ae5f-fd060820ad04 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d5ee1a7-b048-4225-ae5f-fd060820ad04", "name": "5d5ee1a7-b048-4225-ae5f-fd060820ad04", "status": "Failed", "startTime": "2026-08-25T11:02:29.062224891Z", "endTime": "2026-08-25T11:21:35.945858091Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, openshift-route-controller-manager, cluster-storage-operator, ingress-operator, certified-operators-catalog, cluster-version-operator, hosted-cluster-config-operator, openshift-apiserver, community-operators-catalog, cluster-policy-controller, cluster-network-operator, dns-operator, csi-snapshot-controller-operator, catalog-operator, packageserver, cluster-image-registry-operator, olm-operator, cluster-node-tuning-operator, redhat-operators-catalog, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 2 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc000d66520>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-k4pms9xqtc7k" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d5ee1a7-b048-4225-ae5f-fd060820ad04 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d5ee1a7-b048-4225-ae5f-fd060820ad04", "name": "5d5ee1a7-b048-4225-ae5f-fd060820ad04", "status": "Failed", "startTime": "2026-08-25T11:02:29.062224891Z", "endTime": "2026-08-25T11:21:35.945858091Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, openshift-route-controller-manager, cluster-storage-operator, ingress-operator, certified-operators-catalog, cluster-version-operator, hosted-cluster-config-operator, openshift-apiserver, community-operators-catalog, cluster-policy-controller, cluster-network-operator, dns-operator, csi-snapshot-controller-operator, catalog-operator, packageserver, cluster-image-registry-operator, olm-operator, cluster-node-tuning-operator, redhat-operators-catalog, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 2 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-l5fd4zcnxvvv/cluster-candidate-4-20-p4jhqx should provision Unexpected error: <fmt.wrapError \| 0xc001798120>: failed to create HCP cluster cluster-candidate-4-20-p4jhqx: failed waiting for cluster="cluster-candidate-4-20-p4jhqx" in resourcegroup="rg-candidate-4-20-l5fd4zcnxvvv" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0fee2995-90ed-423d-883d-12c988a9efaa -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0fee2995-90ed-423d-883d-12c988a9efaa", "name": "0fee2995-90ed-423d-883d-12c988a9efaa", "status": "Failed", "startTime": "2026-08-25T11:02:28.883200116Z", "endTime": "2026-08-25T11:21:35.924623167Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, ingress-operator, openshift-controller-manager, openshift-route-controller-manager, cluster-storage-operator, redhat-marketplace-catalog, kube-controller-manager, azure-cloud-controller-manager, cluster-network-operator, dns-operator, ignition-server, olm-operator, packageserver, cluster-version-operator, ignition-server-proxy, olm-collect-profiles, cluster-autoscaler, konnectivity-agent, control-plane-pki-operator, openshift-apiserver, redhat-operators-catalog, machine-approver, hosted-cluster-config-operator, cluster-node-tuning-operator, cluster-image-registry-operator, cluster-policy-controller, csi-snapshot-controller-operator, catalog-operator, community-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-l5fd4zcnxvvv/cluster-candidate-4-20-p4jhqx should provision Unexpected error: <fmt.wrapError \| 0xc001798120>: failed to create HCP cluster cluster-candidate-4-20-p4jhqx: failed waiting for cluster="cluster-candidate-4-20-p4jhqx" in resourcegroup="rg-candidate-4-20-l5fd4zcnxvvv" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0fee2995-90ed-423d-883d-12c988a9efaa -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0fee2995-90ed-423d-883d-12c988a9efaa", "name": "0fee2995-90ed-423d-883d-12c988a9efaa", "status": "Failed", "startTime": "2026-08-25T11:02:28.883200116Z", "endTime": "2026-08-25T11:21:35.924623167Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, ingress-operator, openshift-controller-manager, openshift-route-controller-manager, cluster-storage-operator, redhat-marketplace-catalog, kube-controller-manager, azure-cloud-controller-manager, cluster-network-operator, dns-operator, ignition-server, olm-operator, packageserver, cluster-version-operator, ignition-server-proxy, olm-collect-profiles, cluster-autoscaler, konnectivity-agent, control-plane-pki-operator, openshift-apiserver, redhat-operators-catalog, machine-approver, hosted-cluster-config-operator, cluster-node-tuning-operator, cluster-image-registry-operator, cluster-policy-controller, csi-snapshot-controller-operator, catalog-operator, community-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [dns-operator]; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-vmkxjb" with version 4.22.10 on candidate channel Unexpected error: <fmt.wrapError \| 0xc001438440>: failed to create HCP cluster cluster-zstream-4-22-vmkxjb: failed waiting for cluster="cluster-zstream-4-22-vmkxjb" in resourcegroup="rg-zstream-upgrade-4-22-dgzc9xl48h57" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/88ef8722-4c84-4313-a177-16c92648110e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/88ef8722-4c84-4313-a177-16c92648110e", "name": "88ef8722-4c84-4313-a177-16c92648110e", "status": "Failed", "startTime": "2026-08-25T11:02:49.211451249Z", "endTime": "2026-08-25T11:21:55.960423286Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator; hosted cluster degraded: UnavailableReplicas: dns-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-vmkxjb" with version 4.22.10 on candidate channel Unexpected error: <fmt.wrapError \| 0xc001438440>: failed to create HCP cluster cluster-zstream-4-22-vmkxjb: failed waiting for cluster="cluster-zstream-4-22-vmkxjb" in resourcegroup="rg-zstream-upgrade-4-22-dgzc9xl48h57" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/88ef8722-4c84-4313-a177-16c92648110e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/88ef8722-4c84-4313-a177-16c92648110e", "name": "88ef8722-4c84-4313-a177-16c92648110e", "status": "Failed", "startTime": "2026-08-25T11:02:49.211451249Z", "endTime": "2026-08-25T11:21:55.960423286Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator; hosted cluster degraded: UnavailableReplicas: dns-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-zdj2mslnj72d/cluster-candidate-5-0-km49gg should provision Unexpected error: <fmt.wrapError \| 0xc000c0c4a0>: failed to create HCP cluster cluster-candidate-5-0-km49gg: failed waiting for cluster="cluster-candidate-5-0-km49gg" in resourcegroup="rg-candidate-5-0-zdj2mslnj72d" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ecb18d64-1259-4b7c-8758-a10a364e1ea8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ecb18d64-1259-4b7c-8758-a10a364e1ea8", "name": "ecb18d64-1259-4b7c-8758-a10a364e1ea8", "status": "Failed", "startTime": "2026-08-25T11:02:29.10152748Z", "endTime": "2026-08-25T11:21:35.97429848Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, dns-operator, csi-snapshot-controller-operator, community-operators-catalog, openshift-controller-manager, cluster-storage-operator, ingress-operator, olm-operator, cluster-version-operator, cluster-policy-controller, cluster-image-registry-operator, redhat-marketplace-catalog, cluster-network-operator, hosted-cluster-config-operator, cluster-node-tuning-operator, catalog-operator, redhat-operators-catalog, packageserver, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 2 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 2 unavailable replicas, openshift-route-controller-manager deployment has 2 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-zdj2mslnj72d/cluster-candidate-5-0-km49gg should provision Unexpected error: <fmt.wrapError \| 0xc000c0c4a0>: failed to create HCP cluster cluster-candidate-5-0-km49gg: failed waiting for cluster="cluster-candidate-5-0-km49gg" in resourcegroup="rg-candidate-5-0-zdj2mslnj72d" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ecb18d64-1259-4b7c-8758-a10a364e1ea8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ecb18d64-1259-4b7c-8758-a10a364e1ea8", "name": "ecb18d64-1259-4b7c-8758-a10a364e1ea8", "status": "Failed", "startTime": "2026-08-25T11:02:29.10152748Z", "endTime": "2026-08-25T11:21:35.97429848Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, dns-operator, csi-snapshot-controller-operator, community-operators-catalog, openshift-controller-manager, cluster-storage-operator, ingress-operator, olm-operator, cluster-version-operator, cluster-policy-controller, cluster-image-registry-operator, redhat-marketplace-catalog, cluster-network-operator, hosted-cluster-config-operator, cluster-node-tuning-operator, catalog-operator, redhat-operators-catalog, packageserver, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 2 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 2 unavailable replicas, openshift-route-controller-manager deployment has 2 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test Unexpected error: <fmt.wrapError \| 0xc001a82300>: failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-jzcdxxl8w9bb" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f907cfb0-07c7-4e5e-a54f-7c85f0924872 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f907cfb0-07c7-4e5e-a54f-7c85f0924872", "name": "f907cfb0-07c7-4e5e-a54f-7c85f0924872", "status": "Failed", "startTime": "2026-08-25T11:02:39.639932862Z", "endTime": "2026-08-25T11:21:45.83743101Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test Unexpected error: <fmt.wrapError \| 0xc001a82300>: failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-jzcdxxl8w9bb" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f907cfb0-07c7-4e5e-a54f-7c85f0924872 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f907cfb0-07c7-4e5e-a54f-7c85f0924872", "name": "f907cfb0-07c7-4e5e-a54f-7c85f0924872", "status": "Failed", "startTime": "2026-08-25T11:02:39.639932862Z", "endTime": "2026-08-25T11:21:45.83743101Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [dns-operator]; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0xc000f3cae0>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-jfmt6ghkl4r6" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/276ea232-8f07-40f6-a841-fed05291e9a5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/276ea232-8f07-40f6-a841-fed05291e9a5", "name": "276ea232-8f07-40f6-a841-fed05291e9a5", "status": "Failed", "startTime": "2026-08-25T11:02:29.221378186Z", "endTime": "2026-08-25T11:21:35.933589807Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator; hosted cluster degraded: UnavailableReplicas: dns-operator deployment has 2 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0xc000f3cae0>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-jfmt6ghkl4r6" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/276ea232-8f07-40f6-a841-fed05291e9a5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/276ea232-8f07-40f6-a841-fed05291e9a5", "name": "276ea232-8f07-40f6-a841-fed05291e9a5", "status": "Failed", "startTime": "2026-08-25T11:02:29.221378186Z", "endTime": "2026-08-25T11:21:35.933589807Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator; hosted cluster degraded: UnavailableReplicas: dns-operator deployment has 2 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator, ingress-operator, openshift-controller-manager, openshift-route-controller-manager]; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0xc001048a80>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-znbww55lj6vk" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/32f30388-3946-451c-abc4-89ec972b44f5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/32f30388-3946-451c-abc4-89ec972b44f5", "name": "32f30388-3946-451c-abc4-89ec972b44f5", "status": "Failed", "startTime": "2026-08-25T11:02:32.564773496Z", "endTime": "2026-08-25T11:21:36.023308749Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, cluster-network-operator, openshift-controller-manager, ingress-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0xc001048a80>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-znbww55lj6vk" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/32f30388-3946-451c-abc4-89ec972b44f5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/32f30388-3946-451c-abc4-89ec972b44f5", "name": "32f30388-3946-451c-abc4-89ec972b44f5", "status": "Failed", "startTime": "2026-08-25T11:02:32.564773496Z", "endTime": "2026-08-25T11:21:36.023308749Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, cluster-network-operator, openshift-controller-manager, ingress-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred Cluster provisioning failed 10:02:57 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs Unexpected error: <fmt.wrapError \| 0xc001440fe0>: failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-j7x6hnwtqnqh" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0510262-5da5-43b8-8474-f2a1f8d64bb5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0510262-5da5-43b8-8474-f2a1f8d64bb5", "name": "f0510262-5da5-43b8-8474-f2a1f8d64bb5", "status": "Failed", "startTime": "2026-08-25T11:03:11.358657089Z", "endTime": "2026-08-25T11:22:15.937634573Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs Unexpected error: <fmt.wrapError \| 0xc001440fe0>: failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-j7x6hnwtqnqh" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0510262-5da5-43b8-8474-f2a1f8d64bb5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0510262-5da5-43b8-8474-f2a1f8d64bb5", "name": "f0510262-5da5-43b8-8474-f2a1f8d64bb5", "status": "Failed", "startTime": "2026-08-25T11:03:11.358657089Z", "endTime": "2026-08-25T11:22:15.937634573Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000706780>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-jnnzd8dv8sc2" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d4c5630e-4920-46ba-ba8e-4a4b51caf88c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d4c5630e-4920-46ba-ba8e-4a4b51caf88c", "name": "d4c5630e-4920-46ba-ba8e-4a4b51caf88c", "status": "Failed", "startTime": "2026-08-25T11:03:53.022759327Z", "endTime": "2026-08-25T11:22:55.91053294Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000706780>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-jnnzd8dv8sc2" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d4c5630e-4920-46ba-ba8e-4a4b51caf88c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d4c5630e-4920-46ba-ba8e-4a4b51caf88c", "name": "d4c5630e-4920-46ba-ba8e-4a4b51caf88c", "status": "Failed", "startTime": "2026-08-25T11:03:53.022759327Z", "endTime": "2026-08-25T11:22:55.91053294Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, cluster-network-operator, cluster-node-tuning-operator, dns-operator, hosted-cluster-config-operator, ingress-operator, olm-operator, packageserver]; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc000ad6ee0>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-vb2vq2d7pgb7" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9e043488-17a2-4487-83fe-b621f2697284 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9e043488-17a2-4487-83fe-b621f2697284", "name": "9e043488-17a2-4487-83fe-b621f2697284", "status": "Failed", "startTime": "2026-08-25T11:03:38.82687874Z", "endTime": "2026-08-25T11:22:45.901459654Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, hosted-cluster-config-operator, packageserver, cluster-node-tuning-operator, dns-operator, cluster-network-operator, ingress-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc000ad6ee0>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-vb2vq2d7pgb7" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9e043488-17a2-4487-83fe-b621f2697284 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9e043488-17a2-4487-83fe-b621f2697284", "name": "9e043488-17a2-4487-83fe-b621f2697284", "status": "Failed", "startTime": "2026-08-25T11:03:38.82687874Z", "endTime": "2026-08-25T11:22:45.901459654Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, hosted-cluster-config-operator, packageserver, cluster-node-tuning-operator, dns-operator, cluster-network-operator, ingress-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:217]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0xc0010a4120>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-2gtwfxshsqrl" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0b7e1f60-4e89-4cc6-9b44-8aa6ef69312f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0b7e1f60-4e89-4cc6-9b44-8aa6ef69312f", "name": "0b7e1f60-4e89-4cc6-9b44-8aa6ef69312f", "status": "Failed", "startTime": "2026-08-25T11:02:29.28378743Z", "endTime": "2026-08-25T11:21:35.930109254Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:217]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0xc0010a4120>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-2gtwfxshsqrl" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0b7e1f60-4e89-4cc6-9b44-8aa6ef69312f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0b7e1f60-4e89-4cc6-9b44-8aa6ef69312f", "name": "0b7e1f60-4e89-4cc6-9b44-8aa6ef69312f", "status": "Failed", "startTime": "2026-08-25T11:02:29.28378743Z", "endTime": "2026-08-25T11:21:35.930109254Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [dns-operator]; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-w6h87f" with version 4.20.34 on candidate channel Unexpected error: <fmt.wrapError \| 0xc001936140>: failed to create HCP cluster cluster-zstream-4-20-w6h87f: failed waiting for cluster="cluster-zstream-4-20-w6h87f" in resourcegroup="rg-zstream-upgrade-4-20-xmx8p29fpbx4" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a563416d-3979-490a-9710-b74e5cae9107 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a563416d-3979-490a-9710-b74e5cae9107", "name": "a563416d-3979-490a-9710-b74e5cae9107", "status": "Failed", "startTime": "2026-08-25T11:02:29.203167213Z", "endTime": "2026-08-25T11:21:35.888478672Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator; hosted cluster degraded: UnavailableReplicas: dns-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-w6h87f" with version 4.20.34 on candidate channel Unexpected error: <fmt.wrapError \| 0xc001936140>: failed to create HCP cluster cluster-zstream-4-20-w6h87f: failed waiting for cluster="cluster-zstream-4-20-w6h87f" in resourcegroup="rg-zstream-upgrade-4-20-xmx8p29fpbx4" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a563416d-3979-490a-9710-b74e5cae9107 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a563416d-3979-490a-9710-b74e5cae9107", "name": "a563416d-3979-490a-9710-b74e5cae9107", "status": "Failed", "startTime": "2026-08-25T11:02:29.203167213Z", "endTime": "2026-08-25T11:21:35.888478672Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator; hosted cluster degraded: UnavailableReplicas: dns-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator, cluster-network-operator, cluster-node-tuning-operator, cluster-policy-controller, ingress-operator, openshift-controller-manager, openshift-route-controller-manager]; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI Unexpected error: <fmt.wrapError \| 0xc0008526a0>: failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-x8svbm7w8gvv" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/37ed00b3-c155-4497-b144-33a7a3897320 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/37ed00b3-c155-4497-b144-33a7a3897320", "name": "37ed00b3-c155-4497-b144-33a7a3897320", "status": "Failed", "startTime": "2026-08-25T11:02:28.292639561Z", "endTime": "2026-08-25T11:21:35.949263958Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, openshift-controller-manager, cluster-network-operator, cluster-node-tuning-operator, cluster-policy-controller, ingress-operator, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 2 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI Unexpected error: <fmt.wrapError \| 0xc0008526a0>: failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-x8svbm7w8gvv" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/37ed00b3-c155-4497-b144-33a7a3897320 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/37ed00b3-c155-4497-b144-33a7a3897320", "name": "37ed00b3-c155-4497-b144-33a7a3897320", "status": "Failed", "startTime": "2026-08-25T11:02:28.292639561Z", "endTime": "2026-08-25T11:21:35.949263958Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, openshift-controller-manager, cluster-network-operator, cluster-node-tuning-operator, cluster-policy-controller, ingress-operator, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 2 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '20.000000' minutes) Unexpected error: <exported.ResponseError \| 0xc000d9b260>: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/85effd97-ba13-47a3-885a-a4eaa4cf5abc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/85effd97-ba13-47a3-885a-a4eaa4cf5abc", "name": "85effd97-ba13-47a3-885a-a4eaa4cf5abc", "status": "Failed", "startTime": "2026-08-25T11:03:32.916616484Z", "endTime": "2026-08-25T11:22:35.913678576Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '20.000000' minutes) Unexpected error: <exported.ResponseError \| 0xc000d9b260>: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/85effd97-ba13-47a3-885a-a4eaa4cf5abc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/85effd97-ba13-47a3-885a-a4eaa4cf5abc", "name": "85effd97-ba13-47a3-885a-a4eaa4cf5abc", "status": "Failed", "startTime": "2026-08-25T11:03:32.916616484Z", "endTime": "2026-08-25T11:22:35.913678576Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager, catalog-operator, hosted-cluster-config-operator, machine-approver, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 10:02:57 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-bxvq84" with version 4.21.29 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000a00480>: failed to create HCP cluster cluster-zstream-4-21-bxvq84: failed waiting for cluster="cluster-zstream-4-21-bxvq84" in resourcegroup="rg-zstream-upgrade-4-21-f772hgcp6g6p" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c10c2eef-8fa9-4a27-b95a-35876b3db418 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c10c2eef-8fa9-4a27-b95a-35876b3db418", "name": "c10c2eef-8fa9-4a27-b95a-35876b3db418", "status": "Failed", "startTime": "2026-08-25T11:02:29.100638601Z", "endTime": "2026-08-25T11:21:36.009069993Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: certified-operators-catalog, cluster-image-registry-operator, catalog-operator, ingress-operator, csi-snapshot-controller-operator, packageserver, machine-approver, azure-cloud-controller-manager, cluster-network-operator, redhat-marketplace-catalog, cluster-node-tuning-operator, hosted-cluster-config-operator, dns-operator, cluster-policy-controller, redhat-operators-catalog, cluster-version-operator, openshift-controller-manager, olm-operator, community-operators-catalog, openshift-route-controller-manager, cluster-storage-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-bxvq84" with version 4.21.29 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000a00480>: failed to create HCP cluster cluster-zstream-4-21-bxvq84: failed waiting for cluster="cluster-zstream-4-21-bxvq84" in resourcegroup="rg-zstream-upgrade-4-21-f772hgcp6g6p" to finish creating: GET https://rp.j7671680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c10c2eef-8fa9-4a27-b95a-35876b3db418 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c10c2eef-8fa9-4a27-b95a-35876b3db418", "name": "c10c2eef-8fa9-4a27-b95a-35876b3db418", "status": "Failed", "startTime": "2026-08-25T11:02:29.100638601Z", "endTime": "2026-08-25T11:21:36.009069993Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: certified-operators-catalog, cluster-image-registry-operator, catalog-operator, ingress-operator, csi-snapshot-controller-operator, packageserver, machine-approver, azure-cloud-controller-manager, cluster-network-operator, redhat-marketplace-catalog, cluster-node-tuning-operator, hosted-cluster-config-operator, dns-operator, cluster-policy-controller, redhat-operators-catalog, cluster-version-operator, openshift-controller-manager, olm-operator, community-operators-catalog, openshift-route-controller-manager, cluster-storage-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred alert (2) alert [svc] UJClusterProvisionErrors6h30m fired 10:02:57 UTC · failed at alert · test [aro-hcp-observability] [svc] alert UJClusterProvisionErrors6h30m does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-08-25T11:52:38Z Severity: Sev3 Labels: alertname="UJClusterProvisionErrors6h30m", cluster="ci01-j7671680-svc", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7671680/providers/microsoft.monitor/accounts/services-j7671680", severity="3", short_window="30m", slo="cluster-provision-errors" Description: More than 30% of cluster create (install) operations are in failed state sustained over 30 minutes, indicating a medium error budget burn (6x) that would exhaust the 95% SLO budget in ~28 hours. alert [hcp] KubePodNotReady fired 10:02:57 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 171 time(s) (64 unknown, 107 known) Firing 1: State: Resolved Started: 2026-08-25T11:14:21Z Ended: 2026-08-25T11:24:24Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigi4300sn46vldo7trcc7l8m33gfu-j4j6s0z7p6o8r0o", pod="router-86868f6b87-mncnj", severity="warning" Description: Pod ocm-arohcpci01-2sdigi4300sn46vldo7trcc7l8m33gfu-j4j6s0z7p6o8r0o/router-86868f6b87-mncnj has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-08-25T11:14:21Z Ended: 2026-08-25T11:20:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigice0hl86i4gt0ludsc58uuieghd-a3m6i0h0h5c7u0t", pod="etcd-2", severity="warning" Description: Pod ocm-arohcpci01-2sdigice0hl86i4gt0ludsc58uuieghd-a3m6i0h0h5c7u0t/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-08-25T11:15:21Z Ended: 2026-08-25T11:26:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigbl78sli3h6n1b1qhu15o6iecgnp-s2e2g8t3k3c7g2l", pod="cluster-network-operator-5b5ddf888-fcvfd", severity="warning" Description: Pod ocm-arohcpci01-2sdigbl78sli3h6n1b1qhu15o6iecgnp-s2e2g8t3k3c7g2l/cluster-network-operator-5b5ddf888-fcvfd has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-08-25T11:15:21Z Ended: 2026-08-25T11:21:28Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdighajm87paq4cs16pl3ffhosedk96-ea-cluster", pod="kube-apiserver-85c4d7c5b8-n6zkk", severity="warning" Description: Pod ocm-arohcpci01-2sdighajm87paq4cs16pl3ffhosedk96-ea-cluster/kube-apiserver-85c4d7c5b8-n6zkk has been in a non-ready state for longer than 5 minutes. Firing 5: State: Resolved Started: 2026-08-25T11:15:21Z Ended: 2026-08-25T11:21:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigbl78sli3h6n1b1qhu15o6iecgnp-s2e2g8t3k3c7g2l", pod="ingress-operator-5867b47f6c-q64d2", severity="warning" Description: Pod ocm-arohcpci01-2sdigbl78sli3h6n1b1qhu15o6iecgnp-s2e2g8t3k3c7g2l/ingress-operator-5867b47f6c-q64d2 has been in a non-ready state for longer than 5 minutes. Firing 6: State: Resolved Started: 2026-08-25T11:15:21Z Ended: 2026-08-25T11:21:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigbl78sli3h6n1b1qhu15o6iecgnp-s2e2g8t3k3c7g2l", pod="cluster-image-registry-operator-7d55f546b4-98b6x", severity="warning" Description: Pod ocm-arohcpci01-2sdigbl78sli3h6n1b1qhu15o6iecgnp-s2e2g8t3k3c7g2l/cluster-image-registry-operator-7d55f546b4-98b6x has been in a non-ready state for longer than 5 minutes. Firing 7: State: Resolved Started: 2026-08-25T11:15:21Z Ended: 2026-08-25T11:21:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdighp1s7dtgkvt0kn06eru5qhi6hva-i1o3b3q7e1n1e9n", pod="control-plane-operator-5695cc6b8b-cnw6k", severity="warning" Description: Pod ocm-arohcpci01-2sdighp1s7dtgkvt0kn06eru5qhi6hva-i1o3b3q7e1n1e9n/control-plane-operator-5695cc6b8b-cnw6k has been in a non-ready state for longer than 5 minutes. Firing 8: State: Resolved Started: 2026-08-25T11:15:21Z Ended: 2026-08-25T11:21:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdighkj14umms50s1eeb2t0fvtf94n4-z4c8u3h9v6j9o9o", pod="ignition-server-proxy-86df6fdcc8-9k749", severity="warning" Description: Pod ocm-arohcpci01-2sdighkj14umms50s1eeb2t0fvtf94n4-z4c8u3h9v6j9o9o/ignition-server-proxy-86df6fdcc8-9k749 has been in a non-ready state for longer than 5 minutes. Firing 9: State: Resolved Started: 2026-08-25T11:15:21Z Ended: 2026-08-25T11:21:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdighajm87paq4cs16pl3ffhosedk96-ea-cluster", pod="kube-apiserver-85c4d7c5b8-dgxnd", severity="warning" Description: Pod ocm-arohcpci01-2sdighajm87paq4cs16pl3ffhosedk96-ea-cluster/kube-apiserver-85c4d7c5b8-dgxnd has been in a non-ready state for longer than 5 minutes. Firing 10: State: Resolved Started: 2026-08-25T11:15:21Z Ended: 2026-08-25T11:21:29Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigerd9bfd6ciq1hgspbit0kf6oq8j-j3m9k1l7x1h7s8o", pod="cluster-network-operator-7d4d696f5c-d7m5f", severity="warning" Description: Pod ocm-arohcpci01-2sdigerd9bfd6ciq1hgspbit0kf6oq8j-j3m9k1l7x1h7s8o/cluster-network-operator-7d4d696f5c-d7m5f has been in a non-ready state for longer than 5 minutes. Firing 11: State: Resolved Started: 2026-08-25T11:15:21Z Ended: 2026-08-25T11:21:28Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdighkj14umms50s1eeb2t0fvtf94n4-z4c8u3h9v6j9o9o", pod="ignition-server-proxy-86df6fdcc8-pvrr5", severity="warning" Description: Pod ocm-arohcpci01-2sdighkj14umms50s1eeb2t0fvtf94n4-z4c8u3h9v6j9o9o/ignition-server-proxy-86df6fdcc8-pvrr5 has been in a non-ready state for longer than 5 minutes. Firing 12: State: Resolved Started: 2026-08-25T11:15:21Z Ended: 2026-08-25T11:21:28Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigi5nvbb34o4q0i19jdfn9ue3fcf5-c5c3h8c1c5g5f0i", pod="azure-file-csi-driver-operator-865777d9b5-s4lrn", severity="warning" Description: Pod ocm-arohcpci01-2sdigi5nvbb34o4q0i19jdfn9ue3fcf5-c5c3h8c1c5g5f0i/azure-file-csi-driver-operator-865777d9b5-s4lrn has been in a non-ready state for longer than 5 minutes. Firing 13: State: Resolved Started: 2026-08-25T11:15:21Z Ended: 2026-08-25T11:21:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigkfrvlqkgih9nbm82vtk3rdssfk1-w6r1m5h2r9a2f8g", pod="etcd-1", severity="warning" Description: Pod ocm-arohcpci01-2sdigkfrvlqkgih9nbm82vtk3rdssfk1-w6r1m5h2r9a2f8g/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 14: State: Resolved Started: 2026-08-25T11:15:21Z Ended: 2026-08-25T11:21:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig83ue3dh1bfid9jod5bhu1ujp45b-p8k1j3s5d6k3r0m", pod="ingress-operator-5497d8c5f9-976gs", severity="warning" Description: Pod ocm-arohcpci01-2sdig83ue3dh1bfid9jod5bhu1ujp45b-p8k1j3s5d6k3r0m/ingress-operator-5497d8c5f9-976gs has been in a non-ready state for longer than 5 minutes. Firing 15: State: Resolved Started: 2026-08-25T11:15:21Z Ended: 2026-08-25T11:21:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigerd9bfd6ciq1hgspbit0kf6oq8j-j3m9k1l7x1h7s8o", pod="ingress-operator-6b756d9899-tb648", severity="warning" Description: Pod ocm-arohcpci01-2sdigerd9bfd6ciq1hgspbit0kf6oq8j-j3m9k1l7x1h7s8o/ingress-operator-6b756d9899-tb648 has been in a non-ready state for longer than 5 minutes. Firing 16: State: Resolved Started: 2026-08-25T11:16:25Z Ended: 2026-08-25T11:27:22Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihgagmb9aar4uvjhrdttppcgbm57r-agg-cluster", pod="router-64c45976b6-shhnn", severity="warning" Description: Pod ocm-arohcpci01-2sdihgagmb9aar4uvjhrdttppcgbm57r-agg-cluster/router-64c45976b6-shhnn has been in a non-ready state for longer than 5 minutes. Firing 17: State: Resolved Started: 2026-08-25T11:16:25Z Ended: 2026-08-25T11:23:22Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdighba6mofj9027jck00rnhf6mnq12-private-ingress", pod="etcd-0", severity="warning" Description: Pod ocm-arohcpci01-2sdighba6mofj9027jck00rnhf6mnq12-private-ingress/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 18: State: Resolved Started: 2026-08-25T11:16:25Z Ended: 2026-08-25T11:23:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdighbsvc7e80p8g1avr31qr63b8u4e-i6s9l5o7e5m7d5g", pod="etcd-1", severity="warning" Description: Pod ocm-arohcpci01-2sdighbsvc7e80p8g1avr31qr63b8u4e-i6s9l5o7e5m7d5g/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 19: State: Resolved Started: 2026-08-25T11:18:21Z Ended: 2026-08-25T11:24:24Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigs7u0nv00mi79g4kuklicavbrvgq-private-kas", pod="kube-apiserver-7956c4d7f6-nlcp2", severity="warning" Description: Pod ocm-arohcpci01-2sdigs7u0nv00mi79g4kuklicavbrvgq-private-kas/kube-apiserver-7956c4d7f6-nlcp2 has been in a non-ready state for longer than 5 minutes. Firing 20: State: Resolved Started: 2026-08-25T11:18:21Z Ended: 2026-08-25T11:24:25Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigf6q9jsges99tjc5l5b6vs39n2g0-u7k9x9d2b5o4d7r", pod="kube-apiserver-687b9446fb-5rwrw", severity="warning" Description: Pod ocm-arohcpci01-2sdigf6q9jsges99tjc5l5b6vs39n2g0-u7k9x9d2b5o4d7r/kube-apiserver-687b9446fb-5rwrw has been in a non-ready state for longer than 5 minutes. Firing 21: State: Resolved Started: 2026-08-25T11:18:22Z Ended: 2026-08-25T11:25:22Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigjt9hhq647c43sc383r5b05542gn-l9o9j4x2m5k3h8w", pod="cluster-storage-operator-69dbd7db4-m5ws6", severity="warning" Description: Pod ocm-arohcpci01-2sdigjt9hhq647c43sc383r5b05542gn-l9o9j4x2m5k3h8w/cluster-storage-operator-69dbd7db4-m5ws6 has been in a non-ready state for longer than 5 minutes. Firing 22: State: Resolved Started: 2026-08-25T11:18:22Z Ended: 2026-08-25T11:24:24Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigjt9hhq647c43sc383r5b05542gn-l9o9j4x2m5k3h8w", pod="cluster-network-operator-6b85cd7c9c-d5xk4", severity="warning" Description: Pod ocm-arohcpci01-2sdigjt9hhq647c43sc383r5b05542gn-l9o9j4x2m5k3h8w/cluster-network-operator-6b85cd7c9c-d5xk4 has been in a non-ready state for longer than 5 minutes. Firing 23: State: Resolved Started: 2026-08-25T11:18:22Z Ended: 2026-08-25T11:24:25Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigi03ntvea6e3k666eglrm2rhp8id-a2j3u3u2c0q7j5k", pod="openshift-apiserver-7db9b89d69-hfnln", severity="warning" Description: Pod ocm-arohcpci01-2sdigi03ntvea6e3k666eglrm2rhp8id-a2j3u3u2c0q7j5k/openshift-apiserver-7db9b89d69-hfnln has been in a non-ready state for longer than 5 minutes. Firing 24: State: Resolved Started: 2026-08-25T11:18:22Z Ended: 2026-08-25T11:25:22Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigi4topsd1i9raonsubleo01qb8v0-s9s5k7m8c5x0v2w", pod="kube-apiserver-dd9f6c5b7-mrfx2", severity="warning" Description: Pod ocm-arohcpci01-2sdigi4topsd1i9raonsubleo01qb8v0-s9s5k7m8c5x0v2w/kube-apiserver-dd9f6c5b7-mrfx2 has been in a non-ready state for longer than 5 minutes. Firing 25: State: Resolved Started: 2026-08-25T11:18:22Z Ended: 2026-08-25T11:25:22Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdighkj14umms50s1eeb2t0fvtf94n4-z4c8u3h9v6j9o9o", pod="ingress-operator-57bc4cb566-fw8xn", severity="warning" Description: Pod ocm-arohcpci01-2sdighkj14umms50s1eeb2t0fvtf94n4-z4c8u3h9v6j9o9o/ingress-operator-57bc4cb566-fw8xn has been in a non-ready state for longer than 5 minutes. Firing 26: State: Resolved Started: 2026-08-25T11:18:22Z Ended: 2026-08-25T11:24:24Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigjt9hhq647c43sc383r5b05542gn-l9o9j4x2m5k3h8w", pod="ingress-operator-54db749b98-bcptt", severity="warning" Description: Pod ocm-arohcpci01-2sdigjt9hhq647c43sc383r5b05542gn-l9o9j4x2m5k3h8w/ingress-operator-54db749b98-bcptt has been in a non-ready state for longer than 5 minutes. Firing 27: State: Resolved Started: 2026-08-25T11:18:22Z Ended: 2026-08-25T11:24:25Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigi2ogas5reaaof7a0srv70jh6hcg-i4d4z5o5t5w4v5h", pod="cluster-node-tuning-operator-57c96d8565-4s2lb", severity="warning" Description: Pod ocm-arohcpci01-2sdigi2ogas5reaaof7a0srv70jh6hcg-i4d4z5o5t5w4v5h/cluster-node-tuning-operator-57c96d8565-4s2lb has been in a non-ready state for longer than 5 minutes. Firing 28: State: Resolved Started: 2026-08-25T11:18:22Z Ended: 2026-08-25T11:25:23Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigjt9hhq647c43sc383r5b05542gn-l9o9j4x2m5k3h8w", pod="dns-operator-6dd8f6d4dd-g5htt", severity="warning" Description: Pod ocm-arohcpci01-2sdigjt9hhq647c43sc383r5b05542gn-l9o9j4x2m5k3h8w/dns-operator-6dd8f6d4dd-g5htt has been in a non-ready state for longer than 5 minutes. Firing 29: State: Resolved Started: 2026-08-25T11:18:22Z Ended: 2026-08-25T11:25:22Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigs06ckfrn13g1sgge0jvf8sbu0km-l4d8t0p1v0r2j1o", pod="ingress-operator-5c8fb4fbc-6gbrj", severity="warning" Description: Pod ocm-arohcpci01-2sdigs06ckfrn13g1sgge0jvf8sbu0km-l4d8t0p1v0r2j1o/ingress-operator-5c8fb4fbc-6gbrj has been in a non-ready state for longer than 5 minutes. Firing 30: State: Resolved Started: 2026-08-25T11:18:22Z Ended: 2026-08-25T11:25:23Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigs06ckfrn13g1sgge0jvf8sbu0km-l4d8t0p1v0r2j1o", pod="cluster-network-operator-7694545bf9-xvbhj", severity="warning" Description: Pod ocm-arohcpci01-2sdigs06ckfrn13g1sgge0jvf8sbu0km-l4d8t0p1v0r2j1o/cluster-network-operator-7694545bf9-xvbhj has been in a non-ready state for longer than 5 minutes. Firing 31: State: Resolved Started: 2026-08-25T11:18:22Z Ended: 2026-08-25T11:25:23Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdighfolqpd0u9ue3p5c8oft0fg0naq-u5u5r9j6b0w4o1c", pod="kube-apiserver-58bcb897b5-mpp8b", severity="warning" Description: Pod ocm-arohcpci01-2sdighfolqpd0u9ue3p5c8oft0fg0naq-u5u5r9j6b0w4o1c/kube-apiserver-58bcb897b5-mpp8b has been in a non-ready state for longer than 5 minutes. Firing 32: State: Resolved Started: 2026-08-25T11:18:22Z Ended: 2026-08-25T11:25:22Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigs7u0nv00mi79g4kuklicavbrvgq-private-kas", pod="kube-apiserver-7956c4d7f6-rnw6x", severity="warning" Description: Pod ocm-arohcpci01-2sdigs7u0nv00mi79g4kuklicavbrvgq-private-kas/kube-apiserver-7956c4d7f6-rnw6x has been in a non-ready state for longer than 5 minutes. Firing 33: State: Resolved Started: 2026-08-25T11:18:22Z Ended: 2026-08-25T11:25:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdighfolqpd0u9ue3p5c8oft0fg0naq-u5u5r9j6b0w4o1c", pod="kube-apiserver-58bcb897b5-b5rv2", severity="warning" Description: Pod ocm-arohcpci01-2sdighfolqpd0u9ue3p5c8oft0fg0naq-u5u5r9j6b0w4o1c/kube-apiserver-58bcb897b5-b5rv2 has been in a non-ready state for longer than 5 minutes. Firing 34: State: Resolved Started: 2026-08-25T11:18:22Z Ended: 2026-08-25T11:24:25Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigjt9hhq647c43sc383r5b05542gn-l9o9j4x2m5k3h8w", pod="csi-snapshot-controller-operator-76d8c66697-2t55l", severity="warning" Description: Pod ocm-arohcpci01-2sdigjt9hhq647c43sc383r5b05542gn-l9o9j4x2m5k3h8w/csi-snapshot-controller-operator-76d8c66697-2t55l has been in a non-ready state for longer than 5 minutes. Firing 35: State: Resolved Started: 2026-08-25T11:18:22Z Ended: 2026-08-25T11:24:25Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdighcuhd09i09ka2fr8ik662vk6p1p-l2f1w1h5n1u1u8d", pod="cluster-node-tuning-operator-6b7b576bcc-8s87w", severity="warning" Description: Pod ocm-arohcpci01-2sdighcuhd09i09ka2fr8ik662vk6p1p-l2f1w1h5n1u1u8d/cluster-node-tuning-operator-6b7b576bcc-8s87w has been in a non-ready state for longer than 5 minutes. Firing 36: State: Resolved Started: 2026-08-25T11:18:22Z Ended: 2026-08-25T11:25:22Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdighgebj3n8mr39p8ug1auobc5neu0-a1z4t3b4e1i0s3n", pod="kube-apiserver-6c65d59b9b-bv8wx", severity="warning" Description: Pod ocm-arohcpci01-2sdighgebj3n8mr39p8ug1auobc5neu0-a1z4t3b4e1i0s3n/kube-apiserver-6c65d59b9b-bv8wx has been in a non-ready state for longer than 5 minutes. Firing 37: State: Resolved Started: 2026-08-25T11:18:22Z Ended: 2026-08-25T11:25:23Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigjt9hhq647c43sc383r5b05542gn-l9o9j4x2m5k3h8w", pod="dns-operator-867bdbcbc7-gsck8", severity="warning" Description: Pod ocm-arohcpci01-2sdigjt9hhq647c43sc383r5b05542gn-l9o9j4x2m5k3h8w/dns-operator-867bdbcbc7-gsck8 has been in a non-ready state for longer than 5 minutes. Firing 38: State: Resolved Started: 2026-08-25T11:18:22Z Ended: 2026-08-25T11:24:25Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdighoenag5iarnua2ngld4t6dkitqi-t5a4k0j9i8x9d9q", pod="azure-disk-csi-driver-operator-bf5855b64-dpvgt", severity="warning" Description: Pod ocm-arohcpci01-2sdighoenag5iarnua2ngld4t6dkitqi-t5a4k0j9i8x9d9q/azure-disk-csi-driver-operator-bf5855b64-dpvgt has been in a non-ready state for longer than 5 minutes. Firing 39: State: Resolved Started: 2026-08-25T11:19:23Z Ended: 2026-08-25T11:26:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigc9te5976c20animm5nnia7ekeu3-b2f3v8g3z3c8o7r", pod="cluster-storage-operator-7846d67b59-696gz", severity="warning" Description: Pod ocm-arohcpci01-2sdigc9te5976c20animm5nnia7ekeu3-b2f3v8g3z3c8o7r/cluster-storage-operator-7846d67b59-696gz has been in a non-ready state for longer than 5 minutes. Firing 40: State: Resolved Started: 2026-08-25T11:20:28Z Ended: 2026-08-25T11:26:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigi2i4uc1bn8g630if9mr89dd93s6-d3t6h9z1s7g1y0f", pod="multus-admission-controller-574bffdb8c-kkstx", severity="warning" Description: Pod ocm-arohcpci01-2sdigi2i4uc1bn8g630if9mr89dd93s6-d3t6h9z1s7g1y0f/multus-admission-controller-574bffdb8c-kkstx has been in a non-ready state for longer than 5 minutes. Firing 41: State: Resolved Started: 2026-08-25T11:20:28Z Ended: 2026-08-25T11:26:20Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigh8qgenu1ob3k4b4kk5gk82um9oj-m6k9r6v1p6q1p7j", pod="csi-snapshot-controller-operator-9898df5f8-85nl8", severity="warning" Description: Pod ocm-arohcpci01-2sdigh8qgenu1ob3k4b4kk5gk82um9oj-m6k9r6v1p6q1p7j/csi-snapshot-controller-operator-9898df5f8-85nl8 has been in a non-ready state for longer than 5 minutes. Firing 42: State: Resolved Started: 2026-08-25T11:20:29Z Ended: 2026-08-25T11:26:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigh8qgenu1ob3k4b4kk5gk82um9oj-m6k9r6v1p6q1p7j", pod="cluster-network-operator-6d4cf88c9b-n8bgz", severity="warning" Description: Pod ocm-arohcpci01-2sdigh8qgenu1ob3k4b4kk5gk82um9oj-m6k9r6v1p6q1p7j/cluster-network-operator-6d4cf88c9b-n8bgz has been in a non-ready state for longer than 5 minutes. Firing 43: State: Resolved Started: 2026-08-25T11:20:29Z Ended: 2026-08-25T11:26:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig48084ekd4kplpnejldrtd0nto19-f1e9h5t4b7k4k4y", pod="azure-file-csi-driver-operator-76c96977dd-jn44m", severity="warning" Description: Pod ocm-arohcpci01-2sdig48084ekd4kplpnejldrtd0nto19-f1e9h5t4b7k4k4y/azure-file-csi-driver-operator-76c96977dd-jn44m has been in a non-ready state for longer than 5 minutes. Firing 44: State: Resolved Started: 2026-08-25T11:20:29Z Ended: 2026-08-25T11:26:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigmm6ee0hiei23k7aat3sbvnldkgj-z9d9z6o4o8q1n7j", pod="cluster-storage-operator-db6c8d49f-wmv4z", severity="warning" Description: Pod ocm-arohcpci01-2sdigmm6ee0hiei23k7aat3sbvnldkgj-z9d9z6o4o8q1n7j/cluster-storage-operator-db6c8d49f-wmv4z has been in a non-ready state for longer than 5 minutes. Firing 45: State: Resolved Started: 2026-08-25T11:20:29Z Ended: 2026-08-25T11:28:25Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigmm6ee0hiei23k7aat3sbvnldkgj-z9d9z6o4o8q1n7j", pod="dns-operator-b65956b94-mj9mm", severity="warning" Description: Pod ocm-arohcpci01-2sdigmm6ee0hiei23k7aat3sbvnldkgj-z9d9z6o4o8q1n7j/dns-operator-b65956b94-mj9mm has been in a non-ready state for longer than 5 minutes. Firing 46: State: Resolved Started: 2026-08-25T11:20:29Z Ended: 2026-08-25T11:26:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigh8qgenu1ob3k4b4kk5gk82um9oj-m6k9r6v1p6q1p7j", pod="ingress-operator-65bd594575-kvfcs", severity="warning" Description: Pod ocm-arohcpci01-2sdigh8qgenu1ob3k4b4kk5gk82um9oj-m6k9r6v1p6q1p7j/ingress-operator-65bd594575-kvfcs has been in a non-ready state for longer than 5 minutes. Firing 47: State: Resolved Started: 2026-08-25T11:20:29Z Ended: 2026-08-25T11:26:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigmm6ee0hiei23k7aat3sbvnldkgj-z9d9z6o4o8q1n7j", pod="cluster-network-operator-6f4b4758c8-mjgbb", severity="warning" Description: Pod ocm-arohcpci01-2sdigmm6ee0hiei23k7aat3sbvnldkgj-z9d9z6o4o8q1n7j/cluster-network-operator-6f4b4758c8-mjgbb has been in a non-ready state for longer than 5 minutes. Firing 48: State: Resolved Started: 2026-08-25T11:20:30Z Ended: 2026-08-25T11:26:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihgagmb9aar4uvjhrdttppcgbm57r-agg-cluster", pod="kube-apiserver-85774cfd49-vww6s", severity="warning" Description: Pod ocm-arohcpci01-2sdihgagmb9aar4uvjhrdttppcgbm57r-agg-cluster/kube-apiserver-85774cfd49-vww6s has been in a non-ready state for longer than 5 minutes. Firing 49: State: Resolved Started: 2026-08-25T11:20:30Z Ended: 2026-08-25T11:26:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigmm6ee0hiei23k7aat3sbvnldkgj-z9d9z6o4o8q1n7j", pod="csi-snapshot-controller-operator-546cfbb8f6-ffvf4", severity="warning" Description: Pod ocm-arohcpci01-2sdigmm6ee0hiei23k7aat3sbvnldkgj-z9d9z6o4o8q1n7j/csi-snapshot-controller-operator-546cfbb8f6-ffvf4 has been in a non-ready state for longer than 5 minutes. Firing 50: State: Resolved Started: 2026-08-25T11:20:30Z Ended: 2026-08-25T11:26:20Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigmm6ee0hiei23k7aat3sbvnldkgj-z9d9z6o4o8q1n7j", pod="ingress-operator-59c85d587f-259t9", severity="warning" Description: Pod ocm-arohcpci01-2sdigmm6ee0hiei23k7aat3sbvnldkgj-z9d9z6o4o8q1n7j/ingress-operator-59c85d587f-259t9 has been in a non-ready state for longer than 5 minutes. Firing 51: State: Resolved Started: 2026-08-25T11:20:30Z Ended: 2026-08-25T11:28:25Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigh8qgenu1ob3k4b4kk5gk82um9oj-m6k9r6v1p6q1p7j", pod="dns-operator-5444f8f58f-hd42v", severity="warning" Description: Pod ocm-arohcpci01-2sdigh8qgenu1ob3k4b4kk5gk82um9oj-m6k9r6v1p6q1p7j/dns-operator-5444f8f58f-hd42v has been in a non-ready state for longer than 5 minutes. Firing 52: State: Resolved Started: 2026-08-25T11:20:30Z Ended: 2026-08-25T11:26:20Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdih6k5hmsfknhrbnodu51mipr5pjjn-j1v9x5l4l2q6n5w", pod="openshift-apiserver-57b57cd77b-pwtw9", severity="warning" Description: Pod ocm-arohcpci01-2sdih6k5hmsfknhrbnodu51mipr5pjjn-j1v9x5l4l2q6n5w/openshift-apiserver-57b57cd77b-pwtw9 has been in a non-ready state for longer than 5 minutes. Firing 53: State: Resolved Started: 2026-08-25T11:20:30Z Ended: 2026-08-25T11:27:22Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigmm6ee0hiei23k7aat3sbvnldkgj-z9d9z6o4o8q1n7j", pod="cluster-image-registry-operator-f66f8556b-xss96", severity="warning" Description: Pod ocm-arohcpci01-2sdigmm6ee0hiei23k7aat3sbvnldkgj-z9d9z6o4o8q1n7j/cluster-image-registry-operator-f66f8556b-xss96 has been in a non-ready state for longer than 5 minutes. Firing 54: State: Resolved Started: 2026-08-25T11:21:26Z Ended: 2026-08-25T11:40:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdighp1s7dtgkvt0kn06eru5qhi6hva-i1o3b3q7e1n1e9n", pod="router-69dfd7cb55-k56nw", severity="warning" Description: Pod ocm-arohcpci01-2sdighp1s7dtgkvt0kn06eru5qhi6hva-i1o3b3q7e1n1e9n/router-69dfd7cb55-k56nw has been in a non-ready state for longer than 5 minutes. Firing 55: State: Resolved Started: 2026-08-25T11:22:23Z Ended: 2026-08-25T11:28:25Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigkfrvlqkgih9nbm82vtk3rdssfk1-w6r1m5h2r9a2f8g", pod="kube-apiserver-6f78b68b96-b8d5v", severity="warning" Description: Pod ocm-arohcpci01-2sdigkfrvlqkgih9nbm82vtk3rdssfk1-w6r1m5h2r9a2f8g/kube-apiserver-6f78b68b96-b8d5v has been in a non-ready state for longer than 5 minutes. Firing 56: State: Resolved Started: 2026-08-25T11:22:23Z Ended: 2026-08-25T11:28:24Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigi79la932p945fbgrd10a8k2fh8c-m5x1c7l0g1d8v5m", pod="cluster-policy-controller-685d7d6d55-tldwz", severity="warning" Description: Pod ocm-arohcpci01-2sdigi79la932p945fbgrd10a8k2fh8c-m5x1c7l0g1d8v5m/cluster-policy-controller-685d7d6d55-tldwz has been in a non-ready state for longer than 5 minutes. Firing 57: State: Resolved Started: 2026-08-25T11:23:20Z Ended: 2026-08-25T11:29:24Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdighba6mofj9027jck00rnhf6mnq12-private-ingress", pod="openshift-apiserver-6cd99b5b64-jbnrp", severity="warning" Description: Pod ocm-arohcpci01-2sdighba6mofj9027jck00rnhf6mnq12-private-ingress/openshift-apiserver-6cd99b5b64-jbnrp has been in a non-ready state for longer than 5 minutes. Firing 58: State: Resolved Started: 2026-08-25T11:23:20Z Ended: 2026-08-25T11:42:22Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdighave7ee0g1pk0fk8veq8092mk18-f1e9k5d8w7u9q2t", pod="cluster-network-operator-6cbf47c7f6-xthqx", severity="warning" Description: Pod ocm-arohcpci01-2sdighave7ee0g1pk0fk8veq8092mk18-f1e9k5d8w7u9q2t/cluster-network-operator-6cbf47c7f6-xthqx has been in a non-ready state for longer than 5 minutes. Firing 59: State: Resolved Started: 2026-08-25T11:23:20Z Ended: 2026-08-25T11:42:22Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdighave7ee0g1pk0fk8veq8092mk18-f1e9k5d8w7u9q2t", pod="ingress-operator-7766cf748d-t6jch", severity="warning" Description: Pod ocm-arohcpci01-2sdighave7ee0g1pk0fk8veq8092mk18-f1e9k5d8w7u9q2t/ingress-operator-7766cf748d-t6jch has been in a non-ready state for longer than 5 minutes. Firing 60: State: Resolved Started: 2026-08-25T11:23:20Z Ended: 2026-08-25T11:29:24Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdighave7ee0g1pk0fk8veq8092mk18-f1e9k5d8w7u9q2t", pod="cluster-storage-operator-5bd7d48b77-2tpxm", severity="warning" Description: Pod ocm-arohcpci01-2sdighave7ee0g1pk0fk8veq8092mk18-f1e9k5d8w7u9q2t/cluster-storage-operator-5bd7d48b77-2tpxm has been in a non-ready state for longer than 5 minutes. Firing 61: State: Resolved Started: 2026-08-25T11:23:20Z Ended: 2026-08-25T11:42:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdighave7ee0g1pk0fk8veq8092mk18-f1e9k5d8w7u9q2t", pod="dns-operator-54f7c7f776-qtchb", severity="warning" Description: Pod ocm-arohcpci01-2sdighave7ee0g1pk0fk8veq8092mk18-f1e9k5d8w7u9q2t/dns-operator-54f7c7f776-qtchb has been in a non-ready state for longer than 5 minutes. Firing 62: State: Resolved Started: 2026-08-25T11:23:20Z Ended: 2026-08-25T11:29:24Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdighave7ee0g1pk0fk8veq8092mk18-f1e9k5d8w7u9q2t", pod="cluster-version-operator-684b54f49-xdd4x", severity="warning" Description: Pod ocm-arohcpci01-2sdighave7ee0g1pk0fk8veq8092mk18-f1e9k5d8w7u9q2t/cluster-version-operator-684b54f49-xdd4x has been in a non-ready state for longer than 5 minutes. Firing 63: State: Resolved Started: 2026-08-25T11:23:20Z Ended: 2026-08-25T11:29:24Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdighkj14umms50s1eeb2t0fvtf94n4-z4c8u3h9v6j9o9o", pod="multus-admission-controller-5599cbcc84-bxmkb", severity="warning" Description: Pod ocm-arohcpci01-2sdighkj14umms50s1eeb2t0fvtf94n4-z4c8u3h9v6j9o9o/multus-admission-controller-5599cbcc84-bxmkb has been in a non-ready state for longer than 5 minutes. Firing 64: State: Resolved Started: 2026-08-25T11:23:21Z Ended: 2026-08-25T11:29:24Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7671680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdighave7ee0g1pk0fk8veq8092mk18-f1e9k5d8w7u9q2t", pod="csi-snapshot-controller-operator-544df6f47d-rbdp8", severity="warning" Description: Pod ocm-arohcpci01-2sdighave7ee0g1pk0fk8veq8092mk18-f1e9k5d8w7u9q2t/csi-snapshot-controller-operator-544df6f47d-rbdp8 has been in a non-ready state for longer than 5 minutes.
10:02:51	pull-ci-Azure-ARO-HCP-main-e2e-parallel head d5a9f48	alert, e2e	failed	★#6648 (open)	22	e2e (20) ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator, router]; provider Microsoft.RedHatOpenShift 10:02:51 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc000e24460>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-cmpnvt9r42hk" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f79a299e-0c13-4f75-b94f-813c143c5081 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f79a299e-0c13-4f75-b94f-813c143c5081", "name": "f79a299e-0c13-4f75-b94f-813c143c5081", "status": "Failed", "startTime": "2026-08-25T10:58:09.116151219Z", "endTime": "2026-08-25T11:17:17.145229508Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc000e24460>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-cmpnvt9r42hk" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f79a299e-0c13-4f75-b94f-813c143c5081 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f79a299e-0c13-4f75-b94f-813c143c5081", "name": "f79a299e-0c13-4f75-b94f-813c143c5081", "status": "Failed", "startTime": "2026-08-25T10:58:09.116151219Z", "endTime": "2026-08-25T11:17:17.145229508Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator]; provider Microsoft.RedHatOpenShift 10:02:51 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:92]: failed to create HCP cluster "sre-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc00011d6c0>: failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-sbv7dmbdc9np" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d470f9f0-c744-4b72-8591-daf0fd43c83c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d470f9f0-c744-4b72-8591-daf0fd43c83c", "name": "d470f9f0-c744-4b72-8591-daf0fd43c83c", "status": "Failed", "startTime": "2026-08-25T10:58:19.10881324Z", "endTime": "2026-08-25T11:17:27.223455417Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:92]: failed to create HCP cluster "sre-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc00011d6c0>: failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-sbv7dmbdc9np" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d470f9f0-c744-4b72-8591-daf0fd43c83c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d470f9f0-c744-4b72-8591-daf0fd43c83c", "name": "d470f9f0-c744-4b72-8591-daf0fd43c83c", "status": "Failed", "startTime": "2026-08-25T10:58:19.10881324Z", "endTime": "2026-08-25T11:17:27.223455417Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:02:51 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling Unexpected error: <fmt.wrapError \| 0xc001008800>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-gx8j8gn42w5n" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4b44b8a6-930b-4456-b311-c56ef9cad9cc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4b44b8a6-930b-4456-b311-c56ef9cad9cc", "name": "4b44b8a6-930b-4456-b311-c56ef9cad9cc", "status": "Failed", "startTime": "2026-08-25T10:57:58.872615618Z", "endTime": "2026-08-25T11:17:07.212466024Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling Unexpected error: <fmt.wrapError \| 0xc001008800>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-gx8j8gn42w5n" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4b44b8a6-930b-4456-b311-c56ef9cad9cc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4b44b8a6-930b-4456-b311-c56ef9cad9cc", "name": "4b44b8a6-930b-4456-b311-c56ef9cad9cc", "status": "Failed", "startTime": "2026-08-25T10:57:58.872615618Z", "endTime": "2026-08-25T11:17:07.212466024Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 10:02:51 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints Unexpected error: <fmt.wrapError \| 0xc000f19b00>: failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-d9n6fcrmls97" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8e25d7c1-e055-43eb-b802-965b2457d6f2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8e25d7c1-e055-43eb-b802-965b2457d6f2", "name": "8e25d7c1-e055-43eb-b802-965b2457d6f2", "status": "Failed", "startTime": "2026-08-25T11:12:07.690976253Z", "endTime": "2026-08-25T11:31:09.260959759Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints Unexpected error: <fmt.wrapError \| 0xc000f19b00>: failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-d9n6fcrmls97" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8e25d7c1-e055-43eb-b802-965b2457d6f2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8e25d7c1-e055-43eb-b802-965b2457d6f2", "name": "8e25d7c1-e055-43eb-b802-965b2457d6f2", "status": "Failed", "startTime": "2026-08-25T11:12:07.690976253Z", "endTime": "2026-08-25T11:31:09.260959759Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 10:02:51 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:127]: failed to create node pool np-1 for pull secret cluster Unexpected error: <fmt.wrapError \| 0xc000eebc20>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-sbzff66jbsrl" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0db99600-a652-496e-bb1d-7e001a578b70 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0db99600-a652-496e-bb1d-7e001a578b70", "name": "0db99600-a652-496e-bb1d-7e001a578b70", "status": "Failed", "startTime": "2026-08-25T11:10:45.087289014Z", "endTime": "2026-08-25T11:29:49.180304907Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:127]: failed to create node pool np-1 for pull secret cluster Unexpected error: <fmt.wrapError \| 0xc000eebc20>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-sbzff66jbsrl" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0db99600-a652-496e-bb1d-7e001a578b70 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0db99600-a652-496e-bb1d-7e001a578b70", "name": "0db99600-a652-496e-bb1d-7e001a578b70", "status": "Failed", "startTime": "2026-08-25T11:10:45.087289014Z", "endTime": "2026-08-25T11:29:49.180304907Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator, ingress-operator]; provider Microsoft.RedHatOpenShift 10:02:51 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-kj48984pw47r/cluster-candidate-5-0-hjtsmj should provision Unexpected error: <fmt.wrapError \| 0xc0016ea2c0>: failed to create HCP cluster cluster-candidate-5-0-hjtsmj: failed waiting for cluster="cluster-candidate-5-0-hjtsmj" in resourcegroup="rg-candidate-5-0-kj48984pw47r" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/72a51abb-1bf5-4be2-960e-dabd14ac1d8d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/72a51abb-1bf5-4be2-960e-dabd14ac1d8d", "name": "72a51abb-1bf5-4be2-960e-dabd14ac1d8d", "status": "Failed", "startTime": "2026-08-25T10:58:08.811547737Z", "endTime": "2026-08-25T11:17:17.224243602Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-kj48984pw47r/cluster-candidate-5-0-hjtsmj should provision Unexpected error: <fmt.wrapError \| 0xc0016ea2c0>: failed to create HCP cluster cluster-candidate-5-0-hjtsmj: failed waiting for cluster="cluster-candidate-5-0-hjtsmj" in resourcegroup="rg-candidate-5-0-kj48984pw47r" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/72a51abb-1bf5-4be2-960e-dabd14ac1d8d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/72a51abb-1bf5-4be2-960e-dabd14ac1d8d", "name": "72a51abb-1bf5-4be2-960e-dabd14ac1d8d", "status": "Failed", "startTime": "2026-08-25T10:58:08.811547737Z", "endTime": "2026-08-25T11:17:17.224243602Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [ingress-operator]; provider Microsoft.RedHatOpenShift 10:02:51 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260630preview and verify the ingress is internal · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc000fb81a0>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-cj9jxr6srqdd" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a5fe6681-df8c-46f4-b88a-dcdd52ebe331 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a5fe6681-df8c-46f4-b88a-dcdd52ebe331", "name": "a5fe6681-df8c-46f4-b88a-dcdd52ebe331", "status": "Failed", "startTime": "2026-08-25T10:58:18.76965128Z", "endTime": "2026-08-25T11:17:27.213657973Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator; hosted cluster degraded: UnavailableReplicas: ingress-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc000fb81a0>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-cj9jxr6srqdd" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a5fe6681-df8c-46f4-b88a-dcdd52ebe331 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a5fe6681-df8c-46f4-b88a-dcdd52ebe331", "name": "a5fe6681-df8c-46f4-b88a-dcdd52ebe331", "status": "Failed", "startTime": "2026-08-25T10:58:18.76965128Z", "endTime": "2026-08-25T11:17:27.213657973Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator; hosted cluster degraded: UnavailableReplicas: ingress-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator, ingress-operator]; provider Microsoft.RedHatOpenShift 10:02:51 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000f86d60>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-hbpc2f67phzw" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/555056e8-4d19-4e95-b4c8-bcf607cfce34 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/555056e8-4d19-4e95-b4c8-bcf607cfce34", "name": "555056e8-4d19-4e95-b4c8-bcf607cfce34", "status": "Failed", "startTime": "2026-08-25T10:59:29.551776647Z", "endTime": "2026-08-25T11:18:37.175325569Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000f86d60>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-hbpc2f67phzw" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/555056e8-4d19-4e95-b4c8-bcf607cfce34 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/555056e8-4d19-4e95-b4c8-bcf607cfce34", "name": "555056e8-4d19-4e95-b4c8-bcf607cfce34", "status": "Failed", "startTime": "2026-08-25T10:59:29.551776647Z", "endTime": "2026-08-25T11:18:37.175325569Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 10:02:51 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-75n442g7r7zh/cluster-candidate-4-20-t7lsbd should provision Unexpected error: <fmt.wrapError \| 0xc001082120>: failed to create HCP cluster cluster-candidate-4-20-t7lsbd: failed waiting for cluster="cluster-candidate-4-20-t7lsbd" in resourcegroup="rg-candidate-4-20-75n442g7r7zh" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f62eb2c8-ac1a-42b0-a45d-8db7519497d3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f62eb2c8-ac1a-42b0-a45d-8db7519497d3", "name": "f62eb2c8-ac1a-42b0-a45d-8db7519497d3", "status": "Failed", "startTime": "2026-08-25T10:58:19.459653729Z", "endTime": "2026-08-25T11:17:27.222849959Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, hosted-cluster-config-operator, cluster-policy-controller, cluster-storage-operator, dns-operator, ingress-operator, catalog-operator, community-operators-catalog, cluster-version-operator, openshift-controller-manager, cluster-node-tuning-operator, cluster-image-registry-operator, redhat-operators-catalog, certified-operators-catalog, openshift-route-controller-manager, packageserver, olm-operator, redhat-marketplace-catalog, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-75n442g7r7zh/cluster-candidate-4-20-t7lsbd should provision Unexpected error: <fmt.wrapError \| 0xc001082120>: failed to create HCP cluster cluster-candidate-4-20-t7lsbd: failed waiting for cluster="cluster-candidate-4-20-t7lsbd" in resourcegroup="rg-candidate-4-20-75n442g7r7zh" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f62eb2c8-ac1a-42b0-a45d-8db7519497d3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f62eb2c8-ac1a-42b0-a45d-8db7519497d3", "name": "f62eb2c8-ac1a-42b0-a45d-8db7519497d3", "status": "Failed", "startTime": "2026-08-25T10:58:19.459653729Z", "endTime": "2026-08-25T11:17:27.222849959Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, hosted-cluster-config-operator, cluster-policy-controller, cluster-storage-operator, dns-operator, ingress-operator, catalog-operator, community-operators-catalog, cluster-version-operator, openshift-controller-manager, cluster-node-tuning-operator, cluster-image-registry-operator, redhat-operators-catalog, certified-operators-catalog, openshift-route-controller-manager, packageserver, olm-operator, redhat-marketplace-catalog, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred -------------------------------------------------------------------------------- 10:02:51 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:131]: nodepool creation errors: [failed to create NodePool np-one-node: failed waiting for nodepool="np-one-node" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-tp6lxr8bfgv2" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/863038dc-ed8c-4013-be96-35e5dd9a5f59 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/863038dc-ed8c-4013-be96-35e5dd9a5f59", "name": "863038dc-ed8c-4013-be96-35e5dd9a5f59", "status": "Failed", "startTime": "2026-08-25T11:10:58.76533919Z", "endTime": "2026-08-25T11:29:59.215293237Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ] Expected <[]error \| len:1, cap:1>: ... to be empty fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:131]: nodepool creation errors: [failed to create NodePool np-one-node: failed waiting for nodepool="np-one-node" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-tp6lxr8bfgv2" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/863038dc-ed8c-4013-be96-35e5dd9a5f59 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/863038dc-ed8c-4013-be96-35e5dd9a5f59", "name": "863038dc-ed8c-4013-be96-35e5dd9a5f59", "status": "Failed", "startTime": "2026-08-25T11:10:58.76533919Z", "endTime": "2026-08-25T11:29:59.215293237Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ] Expected <[]error \| len:1, cap:1>: ... to be empty ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:02:51 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:442]: failed to create HCP cluster np-skip-minor-zm76p2 at version 4.22.11 Unexpected error: <fmt.wrapError \| 0xc000ed0a80>: failed to create HCP cluster np-skip-minor-zm76p2: failed waiting for cluster="np-skip-minor-zm76p2" in resourcegroup="rg-np-skip-minor-zm76p2-zgjql8zj9727" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/50269510-0da8-4161-a819-8e472daa7784 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/50269510-0da8-4161-a819-8e472daa7784", "name": "50269510-0da8-4161-a819-8e472daa7784", "status": "Failed", "startTime": "2026-08-25T10:58:10.700652772Z", "endTime": "2026-08-25T11:17:17.240136518Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:442]: failed to create HCP cluster np-skip-minor-zm76p2 at version 4.22.11 Unexpected error: <fmt.wrapError \| 0xc000ed0a80>: failed to create HCP cluster np-skip-minor-zm76p2: failed waiting for cluster="np-skip-minor-zm76p2" in resourcegroup="rg-np-skip-minor-zm76p2-zgjql8zj9727" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/50269510-0da8-4161-a819-8e472daa7784 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/50269510-0da8-4161-a819-8e472daa7784", "name": "50269510-0da8-4161-a819-8e472daa7784", "status": "Failed", "startTime": "2026-08-25T10:58:10.700652772Z", "endTime": "2026-08-25T11:17:17.240136518Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 10:02:51 UTC · failed at e2e · test SRE should be able to retrieve serial console logs for a VM · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:324]: failed to create worker nodepool for serial console test Unexpected error: <fmt.wrapError \| 0xc00105d8a0>: failed to create NodePool worker: failed waiting for nodepool="worker" for cluster "sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-gs86r2pvl8xk" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f7fa394b-6357-4697-a2fa-3981a5577b25 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f7fa394b-6357-4697-a2fa-3981a5577b25", "name": "f7fa394b-6357-4697-a2fa-3981a5577b25", "status": "Failed", "startTime": "2026-08-25T11:06:43.537740054Z", "endTime": "2026-08-25T11:25:47.394793135Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:324]: failed to create worker nodepool for serial console test Unexpected error: <fmt.wrapError \| 0xc00105d8a0>: failed to create NodePool worker: failed waiting for nodepool="worker" for cluster "sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-gs86r2pvl8xk" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f7fa394b-6357-4697-a2fa-3981a5577b25 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f7fa394b-6357-4697-a2fa-3981a5577b25", "name": "f7fa394b-6357-4697-a2fa-3981a5577b25", "status": "Failed", "startTime": "2026-08-25T11:06:43.537740054Z", "endTime": "2026-08-25T11:25:47.394793135Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 10:02:51 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc000c79240>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-n6jswvrxrm5x" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4ce3ed2e-b25e-4f6a-b009-44359158a0c0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4ce3ed2e-b25e-4f6a-b009-44359158a0c0", "name": "4ce3ed2e-b25e-4f6a-b009-44359158a0c0", "status": "Failed", "startTime": "2026-08-25T10:59:12.482119209Z", "endTime": "2026-08-25T11:18:17.224952479Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc000c79240>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-n6jswvrxrm5x" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4ce3ed2e-b25e-4f6a-b009-44359158a0c0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4ce3ed2e-b25e-4f6a-b009-44359158a0c0", "name": "4ce3ed2e-b25e-4f6a-b009-44359158a0c0", "status": "Failed", "startTime": "2026-08-25T10:59:12.482119209Z", "endTime": "2026-08-25T11:18:17.224952479Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:02:51 UTC · failed at e2e · test FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc000e2a1a0>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-kg9bjcw5qtdg" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/28b59e86-f017-47aa-9051-f2a88c01f26d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/28b59e86-f017-47aa-9051-f2a88c01f26d", "name": "28b59e86-f017-47aa-9051-f2a88c01f26d", "status": "Failed", "startTime": "2026-08-25T10:58:18.609372122Z", "endTime": "2026-08-25T11:17:27.223023215Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc000e2a1a0>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-kg9bjcw5qtdg" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/28b59e86-f017-47aa-9051-f2a88c01f26d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/28b59e86-f017-47aa-9051-f2a88c01f26d", "name": "28b59e86-f017-47aa-9051-f2a88c01f26d", "status": "Failed", "startTime": "2026-08-25T10:58:18.609372122Z", "endTime": "2026-08-25T11:17:27.223023215Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 10:02:51 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fwpq26" for cluster "cluster-candidate-4-22-fwpq26" Unexpected error: <fmt.wrapError \| 0xc000ee0a00>: failed to create NodePool np-fwpq26: failed waiting for nodepool="np-fwpq26" for cluster "cluster-candidate-4-22-fwpq26" in resourcegroup="rg-candidate-4-22-vdf2b8d7n27r" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1cd59521-c6cb-4009-a07d-22f759f51191 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1cd59521-c6cb-4009-a07d-22f759f51191", "name": "1cd59521-c6cb-4009-a07d-22f759f51191", "status": "Failed", "startTime": "2026-08-25T11:15:37.131944946Z", "endTime": "2026-08-25T11:34:39.183255279Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fwpq26" for cluster "cluster-candidate-4-22-fwpq26" Unexpected error: <fmt.wrapError \| 0xc000ee0a00>: failed to create NodePool np-fwpq26: failed waiting for nodepool="np-fwpq26" for cluster "cluster-candidate-4-22-fwpq26" in resourcegroup="rg-candidate-4-22-vdf2b8d7n27r" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1cd59521-c6cb-4009-a07d-22f759f51191 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1cd59521-c6cb-4009-a07d-22f759f51191", "name": "1cd59521-c6cb-4009-a07d-22f759f51191", "status": "Failed", "startTime": "2026-08-25T11:15:37.131944946Z", "endTime": "2026-08-25T11:34:39.183255279Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:02:51 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-m6ppsj2p4fcx/cluster-candidate-4-21-twphbj should provision Unexpected error: <fmt.wrapError \| 0xc0009ba2e0>: failed to create HCP cluster cluster-candidate-4-21-twphbj: failed waiting for cluster="cluster-candidate-4-21-twphbj" in resourcegroup="rg-candidate-4-21-m6ppsj2p4fcx" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/677d8f28-f8a3-4aa6-9feb-e1997ff2d5a8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/677d8f28-f8a3-4aa6-9feb-e1997ff2d5a8", "name": "677d8f28-f8a3-4aa6-9feb-e1997ff2d5a8", "status": "Failed", "startTime": "2026-08-25T10:58:08.661213076Z", "endTime": "2026-08-25T11:17:17.099289477Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-m6ppsj2p4fcx/cluster-candidate-4-21-twphbj should provision Unexpected error: <fmt.wrapError \| 0xc0009ba2e0>: failed to create HCP cluster cluster-candidate-4-21-twphbj: failed waiting for cluster="cluster-candidate-4-21-twphbj" in resourcegroup="rg-candidate-4-21-m6ppsj2p4fcx" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/677d8f28-f8a3-4aa6-9feb-e1997ff2d5a8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/677d8f28-f8a3-4aa6-9feb-e1997ff2d5a8", "name": "677d8f28-f8a3-4aa6-9feb-e1997ff2d5a8", "status": "Failed", "startTime": "2026-08-25T10:58:08.661213076Z", "endTime": "2026-08-25T11:17:17.099289477Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, packageserver, router]; provider Microsoft.RedHatOpenShift 10:02:51 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc000c85320>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-8h9ptxfrdc67" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0ff2cef3-78a1-4805-a77d-d4ff614f91e4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0ff2cef3-78a1-4805-a77d-d4ff614f91e4", "name": "0ff2cef3-78a1-4805-a77d-d4ff614f91e4", "status": "Failed", "startTime": "2026-08-25T11:00:22.611774831Z", "endTime": "2026-08-25T11:19:27.266054327Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, csi-snapshot-controller-operator, community-operators-catalog, certified-operators-catalog, catalog-operator, olm-operator, openshift-apiserver, cluster-policy-controller, dns-operator, openshift-route-controller-manager, ignition-server-proxy, packageserver, redhat-operators-catalog, control-plane-pki-operator, cluster-version-operator, cluster-network-operator, cluster-node-tuning-operator, konnectivity-agent, redhat-marketplace-catalog, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc000c85320>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-8h9ptxfrdc67" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0ff2cef3-78a1-4805-a77d-d4ff614f91e4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0ff2cef3-78a1-4805-a77d-d4ff614f91e4", "name": "0ff2cef3-78a1-4805-a77d-d4ff614f91e4", "status": "Failed", "startTime": "2026-08-25T11:00:22.611774831Z", "endTime": "2026-08-25T11:19:27.266054327Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, csi-snapshot-controller-operator, community-operators-catalog, certified-operators-catalog, catalog-operator, olm-operator, openshift-apiserver, cluster-policy-controller, dns-operator, openshift-route-controller-manager, ignition-server-proxy, packageserver, redhat-operators-catalog, control-plane-pki-operator, cluster-version-operator, cluster-network-operator, cluster-node-tuning-operator, konnectivity-agent, redhat-marketplace-catalog, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 10:02:51 UTC · failed at e2e · test SRE should return 409 when boot diagnostics is disabled on a VM · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:433]: failed to create worker nodepool for boot diagnostics test Unexpected error: <fmt.wrapError \| 0xc00011d060>: failed to create NodePool worker: failed waiting for nodepool="worker" for cluster "sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-2t5r9x2bsd4t" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c41ebf24-2fbc-424c-afac-9dcffe2ce8a6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c41ebf24-2fbc-424c-afac-9dcffe2ce8a6", "name": "c41ebf24-2fbc-424c-afac-9dcffe2ce8a6", "status": "Failed", "startTime": "2026-08-25T11:09:04.655676542Z", "endTime": "2026-08-25T11:28:09.149731834Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:433]: failed to create worker nodepool for boot diagnostics test Unexpected error: <fmt.wrapError \| 0xc00011d060>: failed to create NodePool worker: failed waiting for nodepool="worker" for cluster "sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-2t5r9x2bsd4t" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c41ebf24-2fbc-424c-afac-9dcffe2ce8a6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c41ebf24-2fbc-424c-afac-9dcffe2ce8a6", "name": "c41ebf24-2fbc-424c-afac-9dcffe2ce8a6", "status": "Failed", "startTime": "2026-08-25T11:09:04.655676542Z", "endTime": "2026-08-25T11:28:09.149731834Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift 10:02:51 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc0008b0740>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-52qjk2wv2ffd" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e88e29b0-abca-45a6-84b3-7185d4bb140b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e88e29b0-abca-45a6-84b3-7185d4bb140b", "name": "e88e29b0-abca-45a6-84b3-7185d4bb140b", "status": "Failed", "startTime": "2026-08-25T10:59:29.34918094Z", "endTime": "2026-08-25T11:18:37.197380239Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc0008b0740>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-52qjk2wv2ffd" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e88e29b0-abca-45a6-84b3-7185d4bb140b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e88e29b0-abca-45a6-84b3-7185d4bb140b", "name": "e88e29b0-abca-45a6-84b3-7185d4bb140b", "status": "Failed", "startTime": "2026-08-25T10:59:29.34918094Z", "endTime": "2026-08-25T11:18:37.197380239Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:02:51 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-b5mv5d" with version 4.21.29 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000def1a0>: failed to create HCP cluster cluster-zstream-4-21-b5mv5d: failed waiting for cluster="cluster-zstream-4-21-b5mv5d" in resourcegroup="rg-zstream-upgrade-4-21-nfcd2w7bxk5w" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d2c78209-32c9-4764-a27b-84232443578b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d2c78209-32c9-4764-a27b-84232443578b", "name": "d2c78209-32c9-4764-a27b-84232443578b", "status": "Failed", "startTime": "2026-08-25T10:57:58.86149776Z", "endTime": "2026-08-25T11:17:07.258943288Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-b5mv5d" with version 4.21.29 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000def1a0>: failed to create HCP cluster cluster-zstream-4-21-b5mv5d: failed waiting for cluster="cluster-zstream-4-21-b5mv5d" in resourcegroup="rg-zstream-upgrade-4-21-nfcd2w7bxk5w" to finish creating: GET https://rp.j8697984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d2c78209-32c9-4764-a27b-84232443578b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d2c78209-32c9-4764-a27b-84232443578b", "name": "d2c78209-32c9-4764-a27b-84232443578b", "status": "Failed", "startTime": "2026-08-25T10:57:58.86149776Z", "endTime": "2026-08-25T11:17:07.258943288Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred alert (2) alert [hcp] KubePodNotReady fired 10:02:51 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 35 time(s) (5 unknown, 30 known) Firing 1: State: Resolved Started: 2026-08-25T11:11:38Z Ended: 2026-08-25T11:30:46Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8697984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdif0lntiqdeb940mhoffbkeopnpu9p-private-kas", pod="router-645d46fdc5-79jdn", severity="warning" Description: Pod ocm-arohcpci01-2sdif0lntiqdeb940mhoffbkeopnpu9p-private-kas/router-645d46fdc5-79jdn has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-08-25T11:12:39Z Ended: 2026-08-25T11:18:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8697984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdif0ma3m6d8mncl4cc474jr39gm8dl-c2x1i0r4y1n2p0o", pod="cluster-network-operator-bf5f5bc79-tq8wx", severity="warning" Description: Pod ocm-arohcpci01-2sdif0ma3m6d8mncl4cc474jr39gm8dl-c2x1i0r4y1n2p0o/cluster-network-operator-bf5f5bc79-tq8wx has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-08-25T11:12:39Z Ended: 2026-08-25T11:18:44Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8697984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdieiveill55dgo37f791mf7ehsfotd-sre-hcp-cluster", pod="ingress-operator-795fd6dfdf-hg5b5", severity="warning" Description: Pod ocm-arohcpci01-2sdieiveill55dgo37f791mf7ehsfotd-sre-hcp-cluster/ingress-operator-795fd6dfdf-hg5b5 has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-08-25T11:12:39Z Ended: 2026-08-25T11:18:42Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8697984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdieiveill55dgo37f791mf7ehsfotd-sre-hcp-cluster", pod="cluster-network-operator-6c9db9877f-dhrmw", severity="warning" Description: Pod ocm-arohcpci01-2sdieiveill55dgo37f791mf7ehsfotd-sre-hcp-cluster/cluster-network-operator-6c9db9877f-dhrmw has been in a non-ready state for longer than 5 minutes. Firing 5: State: Resolved Started: 2026-08-25T11:27:40Z Ended: 2026-08-25T11:33:39Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8697984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdiebjnqq7rr45qo1q02pjcjb34hngm-u1s3f2b9m4y4u9r", pod="network-node-identity-cf5f9c5f6-755bc", severity="warning" Description: Pod ocm-arohcpci01-2sdiebjnqq7rr45qo1q02pjcjb34hngm-u1s3f2b9m4y4u9r/network-node-identity-cf5f9c5f6-755bc has been in a non-ready state for longer than 5 minutes. alert [svc] BackendControllerRetryHotLoop fired 10:02:51 UTC · failed at alert · test [aro-hcp-observability] [svc] alert BackendControllerRetryHotLoop does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-25T11:10:19Z Ended: 2026-08-25T11:46:21Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j8697984-svc", component="backend", name="observemanagedresourcegroup", severity="warning" Description: Backend controller workqueue observemanagedresourcegroup has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work.
10:02:45	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c403afe	alert, e2e	failed	⚠#6638 (open)	48	e2e (45) ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs Unexpected error: <fmt.wrapError \| 0xc000dd5080>: failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-d9wqh8hjrv2l" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ba9a48b3-fad7-429f-8f89-6661e4793c74 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ba9a48b3-fad7-429f-8f89-6661e4793c74", "name": "ba9a48b3-fad7-429f-8f89-6661e4793c74", "status": "Failed", "startTime": "2026-08-25T11:05:23.188516579Z", "endTime": "2026-08-25T11:24:23.029260546Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs Unexpected error: <fmt.wrapError \| 0xc000dd5080>: failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-d9wqh8hjrv2l" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ba9a48b3-fad7-429f-8f89-6661e4793c74 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ba9a48b3-fad7-429f-8f89-6661e4793c74", "name": "ba9a48b3-fad7-429f-8f89-6661e4793c74", "status": "Failed", "startTime": "2026-08-25T11:05:23.188516579Z", "endTime": "2026-08-25T11:24:23.029260546Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator, cluster-policy-controller, cluster-storage-operator, csi-snapshot-controller-operator, dns-operator, ingress-operator, packageserver, router]; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc000936480>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-9ddgb764dmsn" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7cd78ff4-590b-4624-b139-ffc7ac943f90 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7cd78ff4-590b-4624-b139-ffc7ac943f90", "name": "7cd78ff4-590b-4624-b139-ffc7ac943f90", "status": "Failed", "startTime": "2026-08-25T11:04:39.015923741Z", "endTime": "2026-08-25T11:23:43.160352144Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, dns-operator, packageserver, cluster-policy-controller, ingress-operator, olm-operator, cluster-storage-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, packageserver deployment has 2 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc000936480>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-9ddgb764dmsn" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7cd78ff4-590b-4624-b139-ffc7ac943f90 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7cd78ff4-590b-4624-b139-ffc7ac943f90", "name": "7cd78ff4-590b-4624-b139-ffc7ac943f90", "status": "Failed", "startTime": "2026-08-25T11:04:39.015923741Z", "endTime": "2026-08-25T11:23:43.160352144Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, dns-operator, packageserver, cluster-policy-controller, ingress-operator, olm-operator, cluster-storage-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, packageserver deployment has 2 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '20.000000' minutes) Unexpected error: <exported.ResponseError \| 0xc000ef6720>: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a9b5f24b-4624-4d6f-9212-bf4a1fd09040 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a9b5f24b-4624-4d6f-9212-bf4a1fd09040", "name": "a9b5f24b-4624-4d6f-9212-bf4a1fd09040", "status": "Failed", "startTime": "2026-08-25T11:05:22.386559609Z", "endTime": "2026-08-25T11:24:22.932654617Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '20.000000' minutes) Unexpected error: <exported.ResponseError \| 0xc000ef6720>: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a9b5f24b-4624-4d6f-9212-bf4a1fd09040 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a9b5f24b-4624-4d6f-9212-bf4a1fd09040", "name": "a9b5f24b-4624-4d6f-9212-bf4a1fd09040", "status": "Failed", "startTime": "2026-08-25T11:05:22.386559609Z", "endTime": "2026-08-25T11:24:22.932654617Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator, ingress-operator]; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:581]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc00049fd20>: failed to create HCP cluster np-downgrade-gwfjf8: failed waiting for cluster="np-downgrade-gwfjf8" in resourcegroup="rg-np-downgrade-gwfjf8-7kkgmzwhgxk9" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5a2fd7b6-8052-4eb4-a31e-21858150d69e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5a2fd7b6-8052-4eb4-a31e-21858150d69e", "name": "5a2fd7b6-8052-4eb4-a31e-21858150d69e", "status": "Failed", "startTime": "2026-08-25T11:04:40.69418327Z", "endTime": "2026-08-25T11:23:43.19279554Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, cluster-image-registry-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:581]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc00049fd20>: failed to create HCP cluster np-downgrade-gwfjf8: failed waiting for cluster="np-downgrade-gwfjf8" in resourcegroup="rg-np-downgrade-gwfjf8-7kkgmzwhgxk9" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5a2fd7b6-8052-4eb4-a31e-21858150d69e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5a2fd7b6-8052-4eb4-a31e-21858150d69e", "name": "5a2fd7b6-8052-4eb4-a31e-21858150d69e", "status": "Failed", "startTime": "2026-08-25T11:04:40.69418327Z", "endTime": "2026-08-25T11:23:43.19279554Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, cluster-image-registry-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:98]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0xc0012c8120>: failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-4fsszwhdw97q" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/84fe97bd-1239-454a-8374-a673a404e584 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/84fe97bd-1239-454a-8374-a673a404e584", "name": "84fe97bd-1239-454a-8374-a673a404e584", "status": "Failed", "startTime": "2026-08-25T11:02:38.770881546Z", "endTime": "2026-08-25T11:21:43.018048135Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:98]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0xc0012c8120>: failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-4fsszwhdw97q" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/84fe97bd-1239-454a-8374-a673a404e584 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/84fe97bd-1239-454a-8374-a673a404e584", "name": "84fe97bd-1239-454a-8374-a673a404e584", "status": "Failed", "startTime": "2026-08-25T11:02:38.770881546Z", "endTime": "2026-08-25T11:21:43.018048135Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver, router]; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000a1cce0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-994pfg4rbw2z" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f85cfac1-f7cf-4ded-93fb-a1cd604c6c26 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f85cfac1-f7cf-4ded-93fb-a1cd604c6c26", "name": "f85cfac1-f7cf-4ded-93fb-a1cd604c6c26", "status": "Failed", "startTime": "2026-08-25T11:04:18.051164248Z", "endTime": "2026-08-25T11:23:23.080049738Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, konnectivity-agent, catalog-operator, control-plane-pki-operator, openshift-apiserver, ignition-server-proxy, openshift-route-controller-manager, cluster-autoscaler, machine-approver, kube-controller-manager, azure-cloud-controller-manager, redhat-marketplace-catalog, openshift-controller-manager, hosted-cluster-config-operator, olm-operator, cluster-version-operator, cluster-policy-controller, cluster-storage-operator, dns-operator, cluster-image-registry-operator, olm-collect-profiles, certified-operators-catalog, cluster-node-tuning-operator, ingress-operator, packageserver, redhat-operators-catalog, community-operators-catalog, kube-scheduler, ignition-server, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000a1cce0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-994pfg4rbw2z" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f85cfac1-f7cf-4ded-93fb-a1cd604c6c26 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f85cfac1-f7cf-4ded-93fb-a1cd604c6c26", "name": "f85cfac1-f7cf-4ded-93fb-a1cd604c6c26", "status": "Failed", "startTime": "2026-08-25T11:04:18.051164248Z", "endTime": "2026-08-25T11:23:23.080049738Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, konnectivity-agent, catalog-operator, control-plane-pki-operator, openshift-apiserver, ignition-server-proxy, openshift-route-controller-manager, cluster-autoscaler, machine-approver, kube-controller-manager, azure-cloud-controller-manager, redhat-marketplace-catalog, openshift-controller-manager, hosted-cluster-config-operator, olm-operator, cluster-version-operator, cluster-policy-controller, cluster-storage-operator, dns-operator, cluster-image-registry-operator, olm-collect-profiles, certified-operators-catalog, cluster-node-tuning-operator, ingress-operator, packageserver, redhat-operators-catalog, community-operators-catalog, kube-scheduler, ignition-server, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:147]: failed to create HCP cluster "cp-ystream-upgrade-4-21-rvnjl8" at install version 4.20 Unexpected error: <fmt.wrapError \| 0xc000ff8a40>: failed to create HCP cluster cp-ystream-upgrade-4-21-rvnjl8: failed waiting for cluster="cp-ystream-upgrade-4-21-rvnjl8" in resourcegroup="rg-cp-ystream-upgrade-4-21-w4mcqjptjkqr" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/df113e55-476b-4fd3-b5ff-1eee5784babd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/df113e55-476b-4fd3-b5ff-1eee5784babd", "name": "df113e55-476b-4fd3-b5ff-1eee5784babd", "status": "Failed", "startTime": "2026-08-25T11:04:29.586071021Z", "endTime": "2026-08-25T11:23:33.086793566Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, ingress-operator, olm-operator, hosted-cluster-config-operator, cluster-storage-operator, catalog-operator, cluster-network-operator, csi-snapshot-controller-operator, packageserver, cluster-policy-controller, dns-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:147]: failed to create HCP cluster "cp-ystream-upgrade-4-21-rvnjl8" at install version 4.20 Unexpected error: <fmt.wrapError \| 0xc000ff8a40>: failed to create HCP cluster cp-ystream-upgrade-4-21-rvnjl8: failed waiting for cluster="cp-ystream-upgrade-4-21-rvnjl8" in resourcegroup="rg-cp-ystream-upgrade-4-21-w4mcqjptjkqr" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/df113e55-476b-4fd3-b5ff-1eee5784babd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/df113e55-476b-4fd3-b5ff-1eee5784babd", "name": "df113e55-476b-4fd3-b5ff-1eee5784babd", "status": "Failed", "startTime": "2026-08-25T11:04:29.586071021Z", "endTime": "2026-08-25T11:23:33.086793566Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, ingress-operator, olm-operator, hosted-cluster-config-operator, cluster-storage-operator, catalog-operator, cluster-network-operator, csi-snapshot-controller-operator, packageserver, cluster-policy-controller, dns-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:460]: failed to create nodepool nps-420-422 at version 4.20.35 Unexpected error: <fmt.wrapError \| 0xc000990860>: failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-q7nw79" in resourcegroup="rg-np-skip-minor-q7nw79-pbbzs5p6b8hh" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f55038eb-3f9a-4f5d-9582-ceea6e3d3359 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f55038eb-3f9a-4f5d-9582-ceea6e3d3359", "name": "f55038eb-3f9a-4f5d-9582-ceea6e3d3359", "status": "Failed", "startTime": "2026-08-25T11:19:15.692970564Z", "endTime": "2026-08-25T11:38:22.964822621Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:460]: failed to create nodepool nps-420-422 at version 4.20.35 Unexpected error: <fmt.wrapError \| 0xc000990860>: failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-q7nw79" in resourcegroup="rg-np-skip-minor-q7nw79-pbbzs5p6b8hh" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f55038eb-3f9a-4f5d-9582-ceea6e3d3359 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f55038eb-3f9a-4f5d-9582-ceea6e3d3359", "name": "f55038eb-3f9a-4f5d-9582-ceea6e3d3359", "status": "Failed", "startTime": "2026-08-25T11:19:15.692970564Z", "endTime": "2026-08-25T11:38:22.964822621Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-h5hbtn" with version 4.22.10 on candidate channel Unexpected error: <fmt.wrapError \| 0xc0018e6b80>: failed to create HCP cluster cluster-zstream-4-22-h5hbtn: failed waiting for cluster="cluster-zstream-4-22-h5hbtn" in resourcegroup="rg-zstream-upgrade-4-22-6qd4tdr9bh97" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/672f2511-e84b-4f2b-92be-767c90f623e8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/672f2511-e84b-4f2b-92be-767c90f623e8", "name": "672f2511-e84b-4f2b-92be-767c90f623e8", "status": "Failed", "startTime": "2026-08-25T11:04:39.334111057Z", "endTime": "2026-08-25T11:23:43.006484355Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator, ingress-operator, cluster-network-operator, catalog-operator, hosted-cluster-config-operator, cluster-storage-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, cluster-version-operator, olm-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-h5hbtn" with version 4.22.10 on candidate channel Unexpected error: <fmt.wrapError \| 0xc0018e6b80>: failed to create HCP cluster cluster-zstream-4-22-h5hbtn: failed waiting for cluster="cluster-zstream-4-22-h5hbtn" in resourcegroup="rg-zstream-upgrade-4-22-6qd4tdr9bh97" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/672f2511-e84b-4f2b-92be-767c90f623e8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/672f2511-e84b-4f2b-92be-767c90f623e8", "name": "672f2511-e84b-4f2b-92be-767c90f623e8", "status": "Failed", "startTime": "2026-08-25T11:04:39.334111057Z", "endTime": "2026-08-25T11:23:43.006484355Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator, ingress-operator, cluster-network-operator, catalog-operator, hosted-cluster-config-operator, cluster-storage-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, cluster-version-operator, olm-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-dcp4d25dqkhg/cluster-candidate-4-21-798vcj should provision Unexpected error: <fmt.wrapError \| 0xc000983060>: failed to create HCP cluster cluster-candidate-4-21-798vcj: failed waiting for cluster="cluster-candidate-4-21-798vcj" in resourcegroup="rg-candidate-4-21-dcp4d25dqkhg" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8712e933-315a-47e7-9c7b-be7c3ff94f5e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8712e933-315a-47e7-9c7b-be7c3ff94f5e", "name": "8712e933-315a-47e7-9c7b-be7c3ff94f5e", "status": "Failed", "startTime": "2026-08-25T11:02:17.342611915Z", "endTime": "2026-08-25T11:21:23.049204132Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, catalog-operator, olm-operator, dns-operator, packageserver, redhat-operators-catalog, openshift-route-controller-manager, cluster-node-tuning-operator, redhat-marketplace-catalog, cluster-version-operator, cluster-storage-operator, cluster-image-registry-operator, certified-operators-catalog, openshift-controller-manager, community-operators-catalog, hosted-cluster-config-operator, csi-snapshot-controller-operator, cluster-policy-controller, ingress-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 2 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 2 unavailable replicas, openshift-route-controller-manager deployment has 2 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-dcp4d25dqkhg/cluster-candidate-4-21-798vcj should provision Unexpected error: <fmt.wrapError \| 0xc000983060>: failed to create HCP cluster cluster-candidate-4-21-798vcj: failed waiting for cluster="cluster-candidate-4-21-798vcj" in resourcegroup="rg-candidate-4-21-dcp4d25dqkhg" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8712e933-315a-47e7-9c7b-be7c3ff94f5e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8712e933-315a-47e7-9c7b-be7c3ff94f5e", "name": "8712e933-315a-47e7-9c7b-be7c3ff94f5e", "status": "Failed", "startTime": "2026-08-25T11:02:17.342611915Z", "endTime": "2026-08-25T11:21:23.049204132Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, catalog-operator, olm-operator, dns-operator, packageserver, redhat-operators-catalog, openshift-route-controller-manager, cluster-node-tuning-operator, redhat-marketplace-catalog, cluster-version-operator, cluster-storage-operator, cluster-image-registry-operator, certified-operators-catalog, openshift-controller-manager, community-operators-catalog, hosted-cluster-config-operator, csi-snapshot-controller-operator, cluster-policy-controller, ingress-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 2 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 2 unavailable replicas, openshift-route-controller-manager deployment has 2 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:92]: failed to create HCP cluster "sre-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000c14a40>: failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-dzzxfwq4m7h4" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/622e655d-77f3-4820-ba45-6088df8bd8ef -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/622e655d-77f3-4820-ba45-6088df8bd8ef", "name": "622e655d-77f3-4820-ba45-6088df8bd8ef", "status": "Failed", "startTime": "2026-08-25T11:04:38.728472182Z", "endTime": "2026-08-25T11:23:43.188839328Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:92]: failed to create HCP cluster "sre-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000c14a40>: failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-dzzxfwq4m7h4" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/622e655d-77f3-4820-ba45-6088df8bd8ef -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/622e655d-77f3-4820-ba45-6088df8bd8ef", "name": "622e655d-77f3-4820-ba45-6088df8bd8ef", "status": "Failed", "startTime": "2026-08-25T11:04:38.728472182Z", "endTime": "2026-08-25T11:23:43.188839328Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator, cluster-network-operator, dns-operator, ingress-operator, router]; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc000b325c0>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-hcdb8hh7hcl9" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a36ee1da-d072-4335-bca2-f17838bce647 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a36ee1da-d072-4335-bca2-f17838bce647", "name": "a36ee1da-d072-4335-bca2-f17838bce647", "status": "Failed", "startTime": "2026-08-25T11:04:49.267992354Z", "endTime": "2026-08-25T11:23:53.017585492Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, cluster-network-operator, ingress-operator, dns-operator; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc000b325c0>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-hcdb8hh7hcl9" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a36ee1da-d072-4335-bca2-f17838bce647 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a36ee1da-d072-4335-bca2-f17838bce647", "name": "a36ee1da-d072-4335-bca2-f17838bce647", "status": "Failed", "startTime": "2026-08-25T11:04:49.267992354Z", "endTime": "2026-08-25T11:23:53.017585492Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, cluster-network-operator, ingress-operator, dns-operator; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-b2v7l9ll2xj8/cluster-candidate-4-20-k2wzcm should provision Unexpected error: <fmt.wrapError \| 0xc0004a2280>: failed to create HCP cluster cluster-candidate-4-20-k2wzcm: failed waiting for cluster="cluster-candidate-4-20-k2wzcm" in resourcegroup="rg-candidate-4-20-b2v7l9ll2xj8" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9b418ef9-27b0-4dec-abe0-9d11ca69b2b3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9b418ef9-27b0-4dec-abe0-9d11ca69b2b3", "name": "9b418ef9-27b0-4dec-abe0-9d11ca69b2b3", "status": "Failed", "startTime": "2026-08-25T11:03:28.285801063Z", "endTime": "2026-08-25T11:22:33.058166877Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-b2v7l9ll2xj8/cluster-candidate-4-20-k2wzcm should provision Unexpected error: <fmt.wrapError \| 0xc0004a2280>: failed to create HCP cluster cluster-candidate-4-20-k2wzcm: failed waiting for cluster="cluster-candidate-4-20-k2wzcm" in resourcegroup="rg-candidate-4-20-b2v7l9ll2xj8" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9b418ef9-27b0-4dec-abe0-9d11ca69b2b3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9b418ef9-27b0-4dec-abe0-9d11ca69b2b3", "name": "9b418ef9-27b0-4dec-abe0-9d11ca69b2b3", "status": "Failed", "startTime": "2026-08-25T11:03:28.285801063Z", "endTime": "2026-08-25T11:22:33.058166877Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-84b6mc with version 4.21.30 Unexpected error: <fmt.wrapError \| 0xc0013e27e0>: failed to create HCP cluster np-version-upgrade-cluster-84b6mc: failed waiting for cluster="np-version-upgrade-cluster-84b6mc" in resourcegroup="rg-np-version-upgrade-84b6mc-gmpx6gg76fwl" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/982916cf-c7a4-4457-9644-cc6c3c47733f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/982916cf-c7a4-4457-9644-cc6c3c47733f", "name": "982916cf-c7a4-4457-9644-cc6c3c47733f", "status": "Failed", "startTime": "2026-08-25T11:02:39.292750025Z", "endTime": "2026-08-25T11:21:43.032949082Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, ingress-operator, redhat-marketplace-catalog, cluster-version-operator, certified-operators-catalog, openshift-controller-manager, hosted-cluster-config-operator, cluster-storage-operator, olm-operator, community-operators-catalog, dns-operator, csi-snapshot-controller-operator, catalog-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-84b6mc with version 4.21.30 Unexpected error: <fmt.wrapError \| 0xc0013e27e0>: failed to create HCP cluster np-version-upgrade-cluster-84b6mc: failed waiting for cluster="np-version-upgrade-cluster-84b6mc" in resourcegroup="rg-np-version-upgrade-84b6mc-gmpx6gg76fwl" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/982916cf-c7a4-4457-9644-cc6c3c47733f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/982916cf-c7a4-4457-9644-cc6c3c47733f", "name": "982916cf-c7a4-4457-9644-cc6c3c47733f", "status": "Failed", "startTime": "2026-08-25T11:02:39.292750025Z", "endTime": "2026-08-25T11:21:43.032949082Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, ingress-operator, redhat-marketplace-catalog, cluster-version-operator, certified-operators-catalog, openshift-controller-manager, hosted-cluster-config-operator, cluster-storage-operator, olm-operator, community-operators-catalog, dns-operator, csi-snapshot-controller-operator, catalog-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator, ingress-operator]; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-8hkrqg" with version 4.21.29 on candidate channel Unexpected error: <fmt.wrapError \| 0xc0016be120>: failed to create HCP cluster cluster-zstream-4-21-8hkrqg: failed waiting for cluster="cluster-zstream-4-21-8hkrqg" in resourcegroup="rg-zstream-upgrade-4-21-2v84b2nxgnmb" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6021a1ac-f792-4aa4-8f8b-7731245b72da -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6021a1ac-f792-4aa4-8f8b-7731245b72da", "name": "6021a1ac-f792-4aa4-8f8b-7731245b72da", "status": "Failed", "startTime": "2026-08-25T11:04:39.214184097Z", "endTime": "2026-08-25T11:23:43.134206895Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-8hkrqg" with version 4.21.29 on candidate channel Unexpected error: <fmt.wrapError \| 0xc0016be120>: failed to create HCP cluster cluster-zstream-4-21-8hkrqg: failed waiting for cluster="cluster-zstream-4-21-8hkrqg" in resourcegroup="rg-zstream-upgrade-4-21-2v84b2nxgnmb" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6021a1ac-f792-4aa4-8f8b-7731245b72da -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6021a1ac-f792-4aa4-8f8b-7731245b72da", "name": "6021a1ac-f792-4aa4-8f8b-7731245b72da", "status": "Failed", "startTime": "2026-08-25T11:04:39.214184097Z", "endTime": "2026-08-25T11:23:43.134206895Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [dns-operator, ignition-server-proxy, kube-controller-manager]; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc000edefa0>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-rxw4z4c4mjfj" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d7f39717-bb95-4ba8-b531-5600d9435095 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d7f39717-bb95-4ba8-b531-5600d9435095", "name": "d7f39717-bb95-4ba8-b531-5600d9435095", "status": "Failed", "startTime": "2026-08-25T11:03:37.648275001Z", "endTime": "2026-08-25T11:22:43.091522188Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, dns-operator; hosted cluster degraded: UnavailableReplicas: [dns-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc000edefa0>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-rxw4z4c4mjfj" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d7f39717-bb95-4ba8-b531-5600d9435095 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d7f39717-bb95-4ba8-b531-5600d9435095", "name": "d7f39717-bb95-4ba8-b531-5600d9435095", "status": "Failed", "startTime": "2026-08-25T11:03:37.648275001Z", "endTime": "2026-08-25T11:22:43.091522188Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, dns-operator; hosted cluster degraded: UnavailableReplicas: [dns-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0xc000eea240>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-ktb8zbvjkhxv" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8663375f-0c13-4f3e-8ef2-47bc6045ec35 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8663375f-0c13-4f3e-8ef2-47bc6045ec35", "name": "8663375f-0c13-4f3e-8ef2-47bc6045ec35", "status": "Failed", "startTime": "2026-08-25T11:02:37.983795353Z", "endTime": "2026-08-25T11:21:43.01549807Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0xc000eea240>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-ktb8zbvjkhxv" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8663375f-0c13-4f3e-8ef2-47bc6045ec35 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8663375f-0c13-4f3e-8ef2-47bc6045ec35", "name": "8663375f-0c13-4f3e-8ef2-47bc6045ec35", "status": "Failed", "startTime": "2026-08-25T11:02:37.983795353Z", "endTime": "2026-08-25T11:21:43.01549807Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator, cluster-network-operator, csi-snapshot-controller-operator, dns-operator, ingress-operator, packageserver]; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-k6tv8vdvt6xx/cluster-candidate-4-22-zlxjzv should provision Unexpected error: <fmt.wrapError \| 0xc0018feb00>: failed to create HCP cluster cluster-candidate-4-22-zlxjzv: failed waiting for cluster="cluster-candidate-4-22-zlxjzv" in resourcegroup="rg-candidate-4-22-k6tv8vdvt6xx" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7c9dd2d4-422b-44d3-9712-ae568d04d7dd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7c9dd2d4-422b-44d3-9712-ae568d04d7dd", "name": "7c9dd2d4-422b-44d3-9712-ae568d04d7dd", "status": "Failed", "startTime": "2026-08-25T11:04:38.96838764Z", "endTime": "2026-08-25T11:23:43.158826266Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, csi-snapshot-controller-operator, ingress-operator, dns-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-k6tv8vdvt6xx/cluster-candidate-4-22-zlxjzv should provision Unexpected error: <fmt.wrapError \| 0xc0018feb00>: failed to create HCP cluster cluster-candidate-4-22-zlxjzv: failed waiting for cluster="cluster-candidate-4-22-zlxjzv" in resourcegroup="rg-candidate-4-22-k6tv8vdvt6xx" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7c9dd2d4-422b-44d3-9712-ae568d04d7dd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7c9dd2d4-422b-44d3-9712-ae568d04d7dd", "name": "7c9dd2d4-422b-44d3-9712-ae568d04d7dd", "status": "Failed", "startTime": "2026-08-25T11:04:38.96838764Z", "endTime": "2026-08-25T11:23:43.158826266Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, csi-snapshot-controller-operator, ingress-operator, dns-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--rjf7mvrnbkrl/cluster-hs-pre-v6djjz should provision Unexpected error: <fmt.wrapError \| 0xc0009e2440>: failed to create HCP cluster cluster-hs-pre-v6djjz: failed waiting for cluster="cluster-hs-pre-v6djjz" in resourcegroup="rg-hs-pre--rjf7mvrnbkrl" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4553519a-3596-446e-b502-3f9919b48e5a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4553519a-3596-446e-b502-3f9919b48e5a", "name": "4553519a-3596-446e-b502-3f9919b48e5a", "status": "Failed", "startTime": "2026-08-25T11:02:58.208748312Z", "endTime": "2026-08-25T11:22:03.069102239Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, csi-snapshot-controller-operator, cluster-autoscaler, cluster-version-operator, dns-operator, ignition-server, certified-operators-catalog, control-plane-pki-operator, cluster-network-operator, olm-operator, kube-scheduler, hosted-cluster-config-operator, ingress-operator, cluster-image-registry-operator, redhat-marketplace-catalog, openshift-apiserver, konnectivity-agent, redhat-operators-catalog, cluster-node-tuning-operator, olm-collect-profiles, ignition-server-proxy, kube-controller-manager, cluster-policy-controller, azure-cloud-controller-manager, cluster-storage-operator, community-operators-catalog, machine-approver, openshift-controller-manager, catalog-operator, packageserver; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--rjf7mvrnbkrl/cluster-hs-pre-v6djjz should provision Unexpected error: <fmt.wrapError \| 0xc0009e2440>: failed to create HCP cluster cluster-hs-pre-v6djjz: failed waiting for cluster="cluster-hs-pre-v6djjz" in resourcegroup="rg-hs-pre--rjf7mvrnbkrl" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4553519a-3596-446e-b502-3f9919b48e5a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4553519a-3596-446e-b502-3f9919b48e5a", "name": "4553519a-3596-446e-b502-3f9919b48e5a", "status": "Failed", "startTime": "2026-08-25T11:02:58.208748312Z", "endTime": "2026-08-25T11:22:03.069102239Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, csi-snapshot-controller-operator, cluster-autoscaler, cluster-version-operator, dns-operator, ignition-server, certified-operators-catalog, control-plane-pki-operator, cluster-network-operator, olm-operator, kube-scheduler, hosted-cluster-config-operator, ingress-operator, cluster-image-registry-operator, redhat-marketplace-catalog, openshift-apiserver, konnectivity-agent, redhat-operators-catalog, cluster-node-tuning-operator, olm-collect-profiles, ignition-server-proxy, kube-controller-manager, cluster-policy-controller, azure-cloud-controller-manager, cluster-storage-operator, community-operators-catalog, machine-approver, openshift-controller-manager, catalog-operator, packageserver; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, packageserver]; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0xc000fac960>: failed to create HCP cluster ea-listfdq7jq: failed waiting for cluster="ea-listfdq7jq" in resourcegroup="ea-list-pcxkj62k7nc9" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/830cd850-ca4f-4036-ae2b-16567f3cc3fa -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/830cd850-ca4f-4036-ae2b-16567f3cc3fa", "name": "830cd850-ca4f-4036-ae2b-16567f3cc3fa", "status": "Failed", "startTime": "2026-08-25T11:04:49.933938807Z", "endTime": "2026-08-25T11:23:53.031016737Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, dns-operator, catalog-operator, redhat-marketplace-catalog, cluster-network-operator, ingress-operator, csi-snapshot-controller-operator, olm-operator, redhat-operators-catalog, cluster-version-operator, packageserver, openshift-route-controller-manager, certified-operators-catalog, hosted-cluster-config-operator, cluster-image-registry-operator, openshift-controller-manager, cluster-node-tuning-operator, community-operators-catalog, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0xc000fac960>: failed to create HCP cluster ea-listfdq7jq: failed waiting for cluster="ea-listfdq7jq" in resourcegroup="ea-list-pcxkj62k7nc9" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/830cd850-ca4f-4036-ae2b-16567f3cc3fa -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/830cd850-ca4f-4036-ae2b-16567f3cc3fa", "name": "830cd850-ca4f-4036-ae2b-16567f3cc3fa", "status": "Failed", "startTime": "2026-08-25T11:04:49.933938807Z", "endTime": "2026-08-25T11:23:53.031016737Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, dns-operator, catalog-operator, redhat-marketplace-catalog, cluster-network-operator, ingress-operator, csi-snapshot-controller-operator, olm-operator, redhat-operators-catalog, cluster-version-operator, packageserver, openshift-route-controller-manager, certified-operators-catalog, hosted-cluster-config-operator, cluster-image-registry-operator, openshift-controller-manager, cluster-node-tuning-operator, community-operators-catalog, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred Cluster provisioning failed 10:02:45 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-wwr7p9 with version 4.21.30 Unexpected error: <fmt.wrapError \| 0xc0010126e0>: failed to create HCP cluster np-version-upgrade-cluster-wwr7p9: failed waiting for cluster="np-version-upgrade-cluster-wwr7p9" in resourcegroup="rg-np-version-upgrade-wwr7p9-x586qgvl4vx2" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ae3d88bb-54a0-4b93-b6f9-87cac8bac62b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ae3d88bb-54a0-4b93-b6f9-87cac8bac62b", "name": "ae3d88bb-54a0-4b93-b6f9-87cac8bac62b", "status": "Failed", "startTime": "2026-08-25T11:03:30.253062363Z", "endTime": "2026-08-25T11:22:33.132975169Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator, hosted-cluster-config-operator, openshift-controller-manager, cluster-node-tuning-operator, dns-operator, cluster-storage-operator, catalog-operator, cluster-version-operator, cluster-network-operator, csi-snapshot-controller-operator, cluster-policy-controller, olm-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 2 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-wwr7p9 with version 4.21.30 Unexpected error: <fmt.wrapError \| 0xc0010126e0>: failed to create HCP cluster np-version-upgrade-cluster-wwr7p9: failed waiting for cluster="np-version-upgrade-cluster-wwr7p9" in resourcegroup="rg-np-version-upgrade-wwr7p9-x586qgvl4vx2" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ae3d88bb-54a0-4b93-b6f9-87cac8bac62b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ae3d88bb-54a0-4b93-b6f9-87cac8bac62b", "name": "ae3d88bb-54a0-4b93-b6f9-87cac8bac62b", "status": "Failed", "startTime": "2026-08-25T11:03:30.253062363Z", "endTime": "2026-08-25T11:22:33.132975169Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator, hosted-cluster-config-operator, openshift-controller-manager, cluster-node-tuning-operator, dns-operator, cluster-storage-operator, catalog-operator, cluster-version-operator, cluster-network-operator, csi-snapshot-controller-operator, cluster-policy-controller, olm-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 2 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster Unexpected error: <fmt.wrapError \| 0xc000adcd40>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-58qpk5nt74kc" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f05abc28-b92a-4df8-b0e3-a18378ebd5cc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f05abc28-b92a-4df8-b0e3-a18378ebd5cc", "name": "f05abc28-b92a-4df8-b0e3-a18378ebd5cc", "status": "Failed", "startTime": "2026-08-25T11:02:40.088109009Z", "endTime": "2026-08-25T11:21:43.005912277Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster Unexpected error: <fmt.wrapError \| 0xc000adcd40>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-58qpk5nt74kc" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f05abc28-b92a-4df8-b0e3-a18378ebd5cc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f05abc28-b92a-4df8-b0e3-a18378ebd5cc", "name": "f05abc28-b92a-4df8-b0e3-a18378ebd5cc", "status": "Failed", "startTime": "2026-08-25T11:02:40.088109009Z", "endTime": "2026-08-25T11:21:43.005912277Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-node-tuning-operator, cluster-policy-controller, openshift-controller-manager, openshift-route-controller-manager]; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-g4wtj9 with version 4.20.35 Unexpected error: <fmt.wrapError \| 0xc000a8a620>: failed to create HCP cluster np-version-upgrade-cluster-g4wtj9: failed waiting for cluster="np-version-upgrade-cluster-g4wtj9" in resourcegroup="rg-np-version-upgrade-g4wtj9-mrccxbqqnhhc" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7d40e3df-681f-4926-943f-226d7903f59e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7d40e3df-681f-4926-943f-226d7903f59e", "name": "7d40e3df-681f-4926-943f-226d7903f59e", "status": "Failed", "startTime": "2026-08-25T11:04:40.789915832Z", "endTime": "2026-08-25T11:23:43.181882794Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, cluster-policy-controller, cluster-node-tuning-operator, azure-cloud-controller-manager, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 2 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-g4wtj9 with version 4.20.35 Unexpected error: <fmt.wrapError \| 0xc000a8a620>: failed to create HCP cluster np-version-upgrade-cluster-g4wtj9: failed waiting for cluster="np-version-upgrade-cluster-g4wtj9" in resourcegroup="rg-np-version-upgrade-g4wtj9-mrccxbqqnhhc" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7d40e3df-681f-4926-943f-226d7903f59e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7d40e3df-681f-4926-943f-226d7903f59e", "name": "7d40e3df-681f-4926-943f-226d7903f59e", "status": "Failed", "startTime": "2026-08-25T11:04:40.789915832Z", "endTime": "2026-08-25T11:23:43.181882794Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, cluster-policy-controller, cluster-node-tuning-operator, azure-cloud-controller-manager, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 2 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator, cluster-network-operator, dns-operator, ingress-operator]; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-jmx49s" with version 5.0.0-ec.5 on candidate channel Unexpected error: <fmt.wrapError \| 0xc0019b2760>: failed to create HCP cluster cluster-zstream-5-0-jmx49s: failed waiting for cluster="cluster-zstream-5-0-jmx49s" in resourcegroup="rg-zstream-upgrade-5-0-j2n79nx59vfm" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f2fd936b-5571-4d4a-8f06-2d24bc0c10ca -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f2fd936b-5571-4d4a-8f06-2d24bc0c10ca", "name": "f2fd936b-5571-4d4a-8f06-2d24bc0c10ca", "status": "Failed", "startTime": "2026-08-25T11:02:27.608916079Z", "endTime": "2026-08-25T11:21:32.925137069Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator, cluster-network-operator, ingress-operator, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-jmx49s" with version 5.0.0-ec.5 on candidate channel Unexpected error: <fmt.wrapError \| 0xc0019b2760>: failed to create HCP cluster cluster-zstream-5-0-jmx49s: failed waiting for cluster="cluster-zstream-5-0-jmx49s" in resourcegroup="rg-zstream-upgrade-5-0-j2n79nx59vfm" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f2fd936b-5571-4d4a-8f06-2d24bc0c10ca -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f2fd936b-5571-4d4a-8f06-2d24bc0c10ca", "name": "f2fd936b-5571-4d4a-8f06-2d24bc0c10ca", "status": "Failed", "startTime": "2026-08-25T11:02:27.608916079Z", "endTime": "2026-08-25T11:21:32.925137069Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator, cluster-network-operator, ingress-operator, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator, ingress-operator, openshift-route-controller-manager]; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster Unexpected error: <fmt.wrapError \| 0xc001696280>: failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-j578jj9q95nl" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dd6036ee-637e-4111-981e-76356c491fc6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dd6036ee-637e-4111-981e-76356c491fc6", "name": "dd6036ee-637e-4111-981e-76356c491fc6", "status": "Failed", "startTime": "2026-08-25T11:04:39.42538851Z", "endTime": "2026-08-25T11:23:43.113756051Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, openshift-controller-manager, ingress-operator, olm-operator, catalog-operator, packageserver, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster Unexpected error: <fmt.wrapError \| 0xc001696280>: failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-j578jj9q95nl" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dd6036ee-637e-4111-981e-76356c491fc6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dd6036ee-637e-4111-981e-76356c491fc6", "name": "dd6036ee-637e-4111-981e-76356c491fc6", "status": "Failed", "startTime": "2026-08-25T11:04:39.42538851Z", "endTime": "2026-08-25T11:23:43.113756051Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, openshift-controller-manager, ingress-operator, olm-operator, catalog-operator, packageserver, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test SRE can pause schedules to stop backup execution for an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000f0a1e0>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-lfghflc9cww8" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e61d9222-14e5-475a-beb7-5c1acf372d40 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e61d9222-14e5-475a-beb7-5c1acf372d40", "name": "e61d9222-14e5-475a-beb7-5c1acf372d40", "status": "Failed", "startTime": "2026-08-25T11:04:39.98554443Z", "endTime": "2026-08-25T11:23:43.194079743Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, packageserver, cluster-network-operator, dns-operator, csi-snapshot-controller-operator, cluster-version-operator, olm-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000f0a1e0>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-lfghflc9cww8" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e61d9222-14e5-475a-beb7-5c1acf372d40 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e61d9222-14e5-475a-beb7-5c1acf372d40", "name": "e61d9222-14e5-475a-beb7-5c1acf372d40", "status": "Failed", "startTime": "2026-08-25T11:04:39.98554443Z", "endTime": "2026-08-25T11:23:43.194079743Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, packageserver, cluster-network-operator, dns-operator, csi-snapshot-controller-operator, cluster-version-operator, olm-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [ingress-operator]; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test Unexpected error: <fmt.wrapError \| 0xc000ffc140>: failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-ng8s2hdwlfh7" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4a15e78e-de40-4ef2-91b2-32493940aa2b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4a15e78e-de40-4ef2-91b2-32493940aa2b", "name": "4a15e78e-de40-4ef2-91b2-32493940aa2b", "status": "Failed", "startTime": "2026-08-25T11:04:38.979123804Z", "endTime": "2026-08-25T11:23:43.145765165Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator; hosted cluster degraded: UnavailableReplicas: ingress-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test Unexpected error: <fmt.wrapError \| 0xc000ffc140>: failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-ng8s2hdwlfh7" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4a15e78e-de40-4ef2-91b2-32493940aa2b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4a15e78e-de40-4ef2-91b2-32493940aa2b", "name": "4a15e78e-de40-4ef2-91b2-32493940aa2b", "status": "Failed", "startTime": "2026-08-25T11:04:38.979123804Z", "endTime": "2026-08-25T11:23:43.145765165Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator; hosted cluster degraded: UnavailableReplicas: ingress-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0xc000f0e4a0>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-zxfgxgnlxdgl" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8594d25b-0975-46dd-b9ea-b253b264de22 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8594d25b-0975-46dd-b9ea-b253b264de22", "name": "8594d25b-0975-46dd-b9ea-b253b264de22", "status": "Failed", "startTime": "2026-08-25T11:04:30.568853468Z", "endTime": "2026-08-25T11:23:33.079000658Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, cluster-storage-operator, ingress-operator, csi-snapshot-controller-operator, dns-operator, olm-operator, packageserver, hosted-cluster-config-operator, cluster-node-tuning-operator, catalog-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0xc000f0e4a0>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-zxfgxgnlxdgl" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8594d25b-0975-46dd-b9ea-b253b264de22 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8594d25b-0975-46dd-b9ea-b253b264de22", "name": "8594d25b-0975-46dd-b9ea-b253b264de22", "status": "Failed", "startTime": "2026-08-25T11:04:30.568853468Z", "endTime": "2026-08-25T11:23:33.079000658Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, cluster-storage-operator, ingress-operator, csi-snapshot-controller-operator, dns-operator, olm-operator, packageserver, hosted-cluster-config-operator, cluster-node-tuning-operator, catalog-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints Unexpected error: <fmt.wrapError \| 0xc0010f6740>: failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-kl8r766fxv28" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8c7634d2-1f12-4630-a3ca-cc8119d8315e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8c7634d2-1f12-4630-a3ca-cc8119d8315e", "name": "8c7634d2-1f12-4630-a3ca-cc8119d8315e", "status": "Failed", "startTime": "2026-08-25T11:04:40.888667072Z", "endTime": "2026-08-25T11:23:43.024864375Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints Unexpected error: <fmt.wrapError \| 0xc0010f6740>: failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-kl8r766fxv28" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8c7634d2-1f12-4630-a3ca-cc8119d8315e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8c7634d2-1f12-4630-a3ca-cc8119d8315e", "name": "8c7634d2-1f12-4630-a3ca-cc8119d8315e", "status": "Failed", "startTime": "2026-08-25T11:04:40.888667072Z", "endTime": "2026-08-25T11:23:43.024864375Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:88]: failed to create HCP cluster for kusto logs test Unexpected error: <fmt.wrapError \| 0xc00197a700>: failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-m4xxhgr55ksw" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a43ecf60-0976-4b44-9d53-138e3c078e40 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a43ecf60-0976-4b44-9d53-138e3c078e40", "name": "a43ecf60-0976-4b44-9d53-138e3c078e40", "status": "Failed", "startTime": "2026-08-25T11:02:58.779051388Z", "endTime": "2026-08-25T11:22:03.065928952Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:88]: failed to create HCP cluster for kusto logs test Unexpected error: <fmt.wrapError \| 0xc00197a700>: failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-m4xxhgr55ksw" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a43ecf60-0976-4b44-9d53-138e3c078e40 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a43ecf60-0976-4b44-9d53-138e3c078e40", "name": "a43ecf60-0976-4b44-9d53-138e3c078e40", "status": "Failed", "startTime": "2026-08-25T11:02:58.779051388Z", "endTime": "2026-08-25T11:22:03.065928952Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator, dns-operator, ingress-operator]; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc001372260>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-b748z7nmqrc2" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4e67768d-4fd8-4d6f-b652-cd3c57cfd546 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4e67768d-4fd8-4d6f-b652-cd3c57cfd546", "name": "4e67768d-4fd8-4d6f-b652-cd3c57cfd546", "status": "Failed", "startTime": "2026-08-25T11:02:48.323976798Z", "endTime": "2026-08-25T11:21:53.061758244Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, ingress-operator, dns-operator; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc001372260>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-b748z7nmqrc2" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4e67768d-4fd8-4d6f-b652-cd3c57cfd546 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4e67768d-4fd8-4d6f-b652-cd3c57cfd546", "name": "4e67768d-4fd8-4d6f-b652-cd3c57cfd546", "status": "Failed", "startTime": "2026-08-25T11:02:48.323976798Z", "endTime": "2026-08-25T11:21:53.061758244Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, ingress-operator, dns-operator; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager, catalog-operator, hosted-cluster-config-operator, kube-storage-version-migrator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-6qqtw5xzzhgg/cluster-candidate-5-0-skdztd should provision Unexpected error: <fmt.wrapError \| 0xc000ae4220>: failed to create HCP cluster cluster-candidate-5-0-skdztd: failed waiting for cluster="cluster-candidate-5-0-skdztd" in resourcegroup="rg-candidate-5-0-6qqtw5xzzhgg" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd78ff65-5a1c-4595-9723-7a61e4e6ff8e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd78ff65-5a1c-4595-9723-7a61e4e6ff8e", "name": "bd78ff65-5a1c-4595-9723-7a61e4e6ff8e", "status": "Failed", "startTime": "2026-08-25T11:02:58.368910284Z", "endTime": "2026-08-25T11:22:03.073725302Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: azure-cloud-controller-manager, cluster-storage-operator, cluster-version-operator, cluster-policy-controller, csi-snapshot-controller-operator, kube-storage-version-migrator, olm-operator, redhat-marketplace-catalog, cluster-node-tuning-operator, openshift-apiserver, hosted-cluster-config-operator, cluster-network-operator, community-operators-catalog, ingress-operator, catalog-operator, openshift-route-controller-manager, dns-operator, packageserver, redhat-operators-catalog, certified-operators-catalog, cluster-image-registry-operator, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 2 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-6qqtw5xzzhgg/cluster-candidate-5-0-skdztd should provision Unexpected error: <fmt.wrapError \| 0xc000ae4220>: failed to create HCP cluster cluster-candidate-5-0-skdztd: failed waiting for cluster="cluster-candidate-5-0-skdztd" in resourcegroup="rg-candidate-5-0-6qqtw5xzzhgg" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd78ff65-5a1c-4595-9723-7a61e4e6ff8e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd78ff65-5a1c-4595-9723-7a61e4e6ff8e", "name": "bd78ff65-5a1c-4595-9723-7a61e4e6ff8e", "status": "Failed", "startTime": "2026-08-25T11:02:58.368910284Z", "endTime": "2026-08-25T11:22:03.073725302Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: azure-cloud-controller-manager, cluster-storage-operator, cluster-version-operator, cluster-policy-controller, csi-snapshot-controller-operator, kube-storage-version-migrator, olm-operator, redhat-marketplace-catalog, cluster-node-tuning-operator, openshift-apiserver, hosted-cluster-config-operator, cluster-network-operator, community-operators-catalog, ingress-operator, catalog-operator, openshift-route-controller-manager, dns-operator, packageserver, redhat-operators-catalog, certified-operators-catalog, cluster-image-registry-operator, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 2 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [ingress-operator]; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc00151a360>: failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-6hw5fkxk55cz" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c34c7baa-58c7-441d-9b4c-3aa764d9ea35 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c34c7baa-58c7-441d-9b4c-3aa764d9ea35", "name": "c34c7baa-58c7-441d-9b4c-3aa764d9ea35", "status": "Failed", "startTime": "2026-08-25T11:03:29.99287713Z", "endTime": "2026-08-25T11:22:33.093324416Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator; hosted cluster degraded: UnavailableReplicas: ingress-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc00151a360>: failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-6hw5fkxk55cz" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c34c7baa-58c7-441d-9b4c-3aa764d9ea35 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c34c7baa-58c7-441d-9b4c-3aa764d9ea35", "name": "c34c7baa-58c7-441d-9b4c-3aa764d9ea35", "status": "Failed", "startTime": "2026-08-25T11:03:29.99287713Z", "endTime": "2026-08-25T11:22:33.093324416Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator; hosted cluster degraded: UnavailableReplicas: ingress-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test SRE should be able to retrieve serial console logs for a VM · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:308]: failed to create HCP cluster "sre-hcp-cluster-sc" for serial console test Unexpected error: <fmt.wrapError \| 0xc0007fc3a0>: failed to create HCP cluster sre-hcp-cluster-sc: failed waiting for cluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-7qr482mpm7kt" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b7872695-995d-44a9-aff2-649d7b0c3f98 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b7872695-995d-44a9-aff2-649d7b0c3f98", "name": "b7872695-995d-44a9-aff2-649d7b0c3f98", "status": "Failed", "startTime": "2026-08-25T11:04:39.045350362Z", "endTime": "2026-08-25T11:23:42.92215819Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:308]: failed to create HCP cluster "sre-hcp-cluster-sc" for serial console test Unexpected error: <fmt.wrapError \| 0xc0007fc3a0>: failed to create HCP cluster sre-hcp-cluster-sc: failed waiting for cluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-7qr482mpm7kt" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b7872695-995d-44a9-aff2-649d7b0c3f98 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b7872695-995d-44a9-aff2-649d7b0c3f98", "name": "b7872695-995d-44a9-aff2-649d7b0c3f98", "status": "Failed", "startTime": "2026-08-25T11:04:39.045350362Z", "endTime": "2026-08-25T11:23:42.92215819Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator, ingress-operator, router]; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc000b14940>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-56n9dv9dj6vs" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/877ebaca-f2ad-4e6f-9b78-d067abc099c8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/877ebaca-f2ad-4e6f-9b78-d067abc099c8", "name": "877ebaca-f2ad-4e6f-9b78-d067abc099c8", "status": "Failed", "startTime": "2026-08-25T11:05:31.771881865Z", "endTime": "2026-08-25T11:24:32.975906881Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-operator, cluster-network-operator, certified-operators-catalog, ingress-operator, redhat-operators-catalog, redhat-marketplace-catalog, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc000b14940>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-56n9dv9dj6vs" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/877ebaca-f2ad-4e6f-9b78-d067abc099c8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/877ebaca-f2ad-4e6f-9b78-d067abc099c8", "name": "877ebaca-f2ad-4e6f-9b78-d067abc099c8", "status": "Failed", "startTime": "2026-08-25T11:05:31.771881865Z", "endTime": "2026-08-25T11:24:32.975906881Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-operator, cluster-network-operator, certified-operators-catalog, ingress-operator, redhat-operators-catalog, redhat-marketplace-catalog, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc001224260>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-2gp6hm88ddcx" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f13c3733-f010-4f7e-948d-817202b134de -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f13c3733-f010-4f7e-948d-817202b134de", "name": "f13c3733-f010-4f7e-948d-817202b134de", "status": "Failed", "startTime": "2026-08-25T11:05:31.739304195Z", "endTime": "2026-08-25T11:24:32.959602044Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-controller-manager, openshift-route-controller-manager, cluster-storage-operator, catalog-operator, packageserver, kube-scheduler, openshift-apiserver, cluster-node-tuning-operator, csi-snapshot-controller-operator, cluster-autoscaler, openshift-controller-manager, olm-operator, certified-operators-catalog, ingress-operator, konnectivity-agent, machine-approver, cluster-version-operator, olm-collect-profiles, azure-cloud-controller-manager, cluster-network-operator, dns-operator, cluster-image-registry-operator, cluster-policy-controller, hosted-cluster-config-operator, kube-storage-version-migrator, redhat-operators-catalog, community-operators-catalog, control-plane-pki-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc001224260>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-2gp6hm88ddcx" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f13c3733-f010-4f7e-948d-817202b134de -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f13c3733-f010-4f7e-948d-817202b134de", "name": "f13c3733-f010-4f7e-948d-817202b134de", "status": "Failed", "startTime": "2026-08-25T11:05:31.739304195Z", "endTime": "2026-08-25T11:24:32.959602044Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-controller-manager, openshift-route-controller-manager, cluster-storage-operator, catalog-operator, packageserver, kube-scheduler, openshift-apiserver, cluster-node-tuning-operator, csi-snapshot-controller-operator, cluster-autoscaler, openshift-controller-manager, olm-operator, certified-operators-catalog, ingress-operator, konnectivity-agent, machine-approver, cluster-version-operator, olm-collect-profiles, azure-cloud-controller-manager, cluster-network-operator, dns-operator, cluster-image-registry-operator, cluster-policy-controller, hosted-cluster-config-operator, kube-storage-version-migrator, redhat-operators-catalog, community-operators-catalog, control-plane-pki-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128" Unexpected error: <fmt.wrapError \| 0xc001336240>: failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-2zg2mjdfgmzm" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/489540be-0a79-4401-90e2-c83238026963 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/489540be-0a79-4401-90e2-c83238026963", "name": "489540be-0a79-4401-90e2-c83238026963", "status": "Failed", "startTime": "2026-08-25T11:04:39.053775952Z", "endTime": "2026-08-25T11:23:43.202927029Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, certified-operators-catalog, cluster-node-tuning-operator, csi-snapshot-controller-operator, community-operators-catalog, openshift-route-controller-manager, ingress-operator, openshift-controller-manager, cluster-policy-controller, cluster-version-operator, hosted-cluster-config-operator, catalog-operator, redhat-marketplace-catalog, cluster-storage-operator, dns-operator, cluster-network-operator, cluster-image-registry-operator, olm-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128" Unexpected error: <fmt.wrapError \| 0xc001336240>: failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-2zg2mjdfgmzm" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/489540be-0a79-4401-90e2-c83238026963 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/489540be-0a79-4401-90e2-c83238026963", "name": "489540be-0a79-4401-90e2-c83238026963", "status": "Failed", "startTime": "2026-08-25T11:04:39.053775952Z", "endTime": "2026-08-25T11:23:43.202927029Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, certified-operators-catalog, cluster-node-tuning-operator, csi-snapshot-controller-operator, community-operators-catalog, openshift-route-controller-manager, ingress-operator, openshift-controller-manager, cluster-policy-controller, cluster-version-operator, hosted-cluster-config-operator, catalog-operator, redhat-marketplace-catalog, cluster-storage-operator, dns-operator, cluster-network-operator, cluster-image-registry-operator, olm-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager, catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260630preview and verify the ingress is internal · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc00134ebc0>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-7p6xhpjkj2d7" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e9b87905-c782-40c9-95b4-fab4bb1cb211 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e9b87905-c782-40c9-95b4-fab4bb1cb211", "name": "e9b87905-c782-40c9-95b4-fab4bb1cb211", "status": "Failed", "startTime": "2026-08-25T11:04:49.498742711Z", "endTime": "2026-08-25T11:23:52.96330385Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, ingress-operator, cluster-policy-controller, dns-operator, catalog-operator, openshift-route-controller-manager, cluster-node-tuning-operator, redhat-operators-catalog, cluster-storage-operator, cluster-network-operator, cluster-image-registry-operator, packageserver, azure-cloud-controller-manager, openshift-controller-manager, certified-operators-catalog, olm-operator, redhat-marketplace-catalog, hosted-cluster-config-operator, csi-snapshot-controller-operator, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc00134ebc0>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-7p6xhpjkj2d7" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e9b87905-c782-40c9-95b4-fab4bb1cb211 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e9b87905-c782-40c9-95b4-fab4bb1cb211", "name": "e9b87905-c782-40c9-95b4-fab4bb1cb211", "status": "Failed", "startTime": "2026-08-25T11:04:49.498742711Z", "endTime": "2026-08-25T11:23:52.96330385Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, ingress-operator, cluster-policy-controller, dns-operator, catalog-operator, openshift-route-controller-manager, cluster-node-tuning-operator, redhat-operators-catalog, cluster-storage-operator, cluster-network-operator, cluster-image-registry-operator, packageserver, azure-cloud-controller-manager, openshift-controller-manager, certified-operators-catalog, olm-operator, redhat-marketplace-catalog, hosted-cluster-config-operator, csi-snapshot-controller-operator, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator]; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-ssd9kv" with version 4.20.34 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000a661a0>: failed to create HCP cluster cluster-zstream-4-20-ssd9kv: failed waiting for cluster="cluster-zstream-4-20-ssd9kv" in resourcegroup="rg-zstream-upgrade-4-20-9qqpqzsl79cn" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a401f1f7-37d5-4c48-9273-6d48eff0c7a0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a401f1f7-37d5-4c48-9273-6d48eff0c7a0", "name": "a401f1f7-37d5-4c48-9273-6d48eff0c7a0", "status": "Failed", "startTime": "2026-08-25T11:04:38.959728228Z", "endTime": "2026-08-25T11:23:43.108250186Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-ssd9kv" with version 4.20.34 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000a661a0>: failed to create HCP cluster cluster-zstream-4-20-ssd9kv: failed waiting for cluster="cluster-zstream-4-20-ssd9kv" in resourcegroup="rg-zstream-upgrade-4-20-9qqpqzsl79cn" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a401f1f7-37d5-4c48-9273-6d48eff0c7a0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a401f1f7-37d5-4c48-9273-6d48eff0c7a0", "name": "a401f1f7-37d5-4c48-9273-6d48eff0c7a0", "status": "Failed", "startTime": "2026-08-25T11:04:38.959728228Z", "endTime": "2026-08-25T11:23:43.108250186Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [openshift-controller-manager, openshift-route-controller-manager]; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc000e26600>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-g94rgnmwffff" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00f455ba-7a0f-4042-81b1-6b4e1d717422 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00f455ba-7a0f-4042-81b1-6b4e1d717422", "name": "00f455ba-7a0f-4042-81b1-6b4e1d717422", "status": "Failed", "startTime": "2026-08-25T11:02:28.361104Z", "endTime": "2026-08-25T11:21:33.042291432Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc000e26600>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-g94rgnmwffff" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00f455ba-7a0f-4042-81b1-6b4e1d717422 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00f455ba-7a0f-4042-81b1-6b4e1d717422", "name": "00f455ba-7a0f-4042-81b1-6b4e1d717422", "status": "Failed", "startTime": "2026-08-25T11:02:28.361104Z", "endTime": "2026-08-25T11:21:33.042291432Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator, csi-snapshot-controller-operator, dns-operator, ingress-operator]; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:302]: failed to create HCP cluster np-noedge-j4v9fp Unexpected error: <fmt.wrapError \| 0xc000eaa980>: failed to create HCP cluster np-noedge-j4v9fp: failed waiting for cluster="np-noedge-j4v9fp" in resourcegroup="rg-np-noedge-j4v9fp-4r7jnknp4pwv" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e41d1c34-8f1a-4c5a-b4ca-387c7744ff7c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e41d1c34-8f1a-4c5a-b4ca-387c7744ff7c", "name": "e41d1c34-8f1a-4c5a-b4ca-387c7744ff7c", "status": "Failed", "startTime": "2026-08-25T11:04:30.442745031Z", "endTime": "2026-08-25T11:23:33.077388395Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, ingress-operator, csi-snapshot-controller-operator, dns-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:302]: failed to create HCP cluster np-noedge-j4v9fp Unexpected error: <fmt.wrapError \| 0xc000eaa980>: failed to create HCP cluster np-noedge-j4v9fp: failed waiting for cluster="np-noedge-j4v9fp" in resourcegroup="rg-np-noedge-j4v9fp-4r7jnknp4pwv" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e41d1c34-8f1a-4c5a-b4ca-387c7744ff7c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e41d1c34-8f1a-4c5a-b4ca-387c7744ff7c", "name": "e41d1c34-8f1a-4c5a-b4ca-387c7744ff7c", "status": "Failed", "startTime": "2026-08-25T11:04:30.442745031Z", "endTime": "2026-08-25T11:23:33.077388395Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, ingress-operator, csi-snapshot-controller-operator, dns-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:217]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0xc000df13c0>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-hf9vt4xl7cvh" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a2c0e2a4-0c60-4344-aad1-28e0efa81dc3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a2c0e2a4-0c60-4344-aad1-28e0efa81dc3", "name": "a2c0e2a4-0c60-4344-aad1-28e0efa81dc3", "status": "Failed", "startTime": "2026-08-25T11:02:58.592272694Z", "endTime": "2026-08-25T11:22:03.020998968Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, cluster-storage-operator, ingress-operator, community-operators-catalog, csi-snapshot-controller-operator, catalog-operator, redhat-marketplace-catalog, certified-operators-catalog, dns-operator, redhat-operators-catalog, olm-operator, cluster-network-operator, hosted-cluster-config-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:217]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0xc000df13c0>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-hf9vt4xl7cvh" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a2c0e2a4-0c60-4344-aad1-28e0efa81dc3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a2c0e2a4-0c60-4344-aad1-28e0efa81dc3", "name": "a2c0e2a4-0c60-4344-aad1-28e0efa81dc3", "status": "Failed", "startTime": "2026-08-25T11:02:58.592272694Z", "endTime": "2026-08-25T11:22:03.020998968Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, cluster-storage-operator, ingress-operator, community-operators-catalog, csi-snapshot-controller-operator, catalog-operator, redhat-marketplace-catalog, certified-operators-catalog, dns-operator, redhat-operators-catalog, olm-operator, cluster-network-operator, hosted-cluster-config-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator]; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test Unexpected error: <fmt.wrapError \| 0xc000eb0120>: failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-bntb7vm54fqr" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9f4a7238-dfd3-42b2-8b5c-10dd1b445fb3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9f4a7238-dfd3-42b2-8b5c-10dd1b445fb3", "name": "9f4a7238-dfd3-42b2-8b5c-10dd1b445fb3", "status": "Failed", "startTime": "2026-08-25T11:02:27.301468987Z", "endTime": "2026-08-25T11:21:32.988392269Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator; hosted cluster degraded: UnavailableReplicas: cluster-network-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test Unexpected error: <fmt.wrapError \| 0xc000eb0120>: failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-bntb7vm54fqr" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9f4a7238-dfd3-42b2-8b5c-10dd1b445fb3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9f4a7238-dfd3-42b2-8b5c-10dd1b445fb3", "name": "9f4a7238-dfd3-42b2-8b5c-10dd1b445fb3", "status": "Failed", "startTime": "2026-08-25T11:02:27.301468987Z", "endTime": "2026-08-25T11:21:32.988392269Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator; hosted cluster degraded: UnavailableReplicas: cluster-network-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [csi-snapshot-controller-operator, dns-operator, ignition-server-proxy, packageserver]; provider Microsoft.RedHatOpenShift 10:02:45 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: failed to create HCP cluster np-update-nodes-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000a8c220>: failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-c2x56bx2tvvb" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8707f304-4ff5-45ff-8354-0739ca14d80f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8707f304-4ff5-45ff-8354-0739ca14d80f", "name": "8707f304-4ff5-45ff-8354-0739ca14d80f", "status": "Failed", "startTime": "2026-08-25T11:03:09.304723426Z", "endTime": "2026-08-25T11:22:13.012989469Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, dns-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, packageserver deployment has 2 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: failed to create HCP cluster np-update-nodes-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000a8c220>: failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-c2x56bx2tvvb" to finish creating: GET https://rp.j5488384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8707f304-4ff5-45ff-8354-0739ca14d80f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8707f304-4ff5-45ff-8354-0739ca14d80f", "name": "8707f304-4ff5-45ff-8354-0739ca14d80f", "status": "Failed", "startTime": "2026-08-25T11:03:09.304723426Z", "endTime": "2026-08-25T11:22:13.012989469Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, dns-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, packageserver deployment has 2 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred alert (3) alert [svc] UJClusterProvisionErrors1h5m fired 10:02:45 UTC · failed at alert · test [aro-hcp-observability] [svc] alert UJClusterProvisionErrors1h5m does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-08-25T11:29:19Z Severity: Sev3 Labels: alertname="UJClusterProvisionErrors1h5m", cluster="ci01-j5488384-svc", component="slo", long_window="1h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5488384/providers/microsoft.monitor/accounts/services-j5488384", severity="3", short_window="5m", slo="cluster-provision-errors" Description: More than 72% of cluster create (install) operations are in failed state, indicating a fast error budget burn (14.4x) that would exhaust the 95% SLO budget in ~12 hours. A regional install failure of this magnitude typically points at a shared dependency (e.g. registry, DNS, or ARM) rather than individual clusters. alert [hcp] KubePodNotReady fired 10:02:45 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 208 time(s) (81 unknown, 127 known) Firing 1: State: Resolved Started: 2026-08-25T11:14:44Z Ended: 2026-08-25T11:21:44Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihfurlbg9kr3lc200gkudg3qrsi33-f5k4p3y8a6g9u0l", pod="etcd-1", severity="warning" Description: Pod ocm-arohcpci01-2sdihfurlbg9kr3lc200gkudg3qrsi33-f5k4p3y8a6g9u0l/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-08-25T11:14:44Z Ended: 2026-08-25T11:21:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihhr5bq5su1ium6nlh4545uep5uje-basic-cluster", pod="etcd-2", severity="warning" Description: Pod ocm-arohcpci01-2sdihhr5bq5su1ium6nlh4545uep5uje-basic-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-08-25T11:14:44Z Ended: 2026-08-25T11:23:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigjqmlu1hialupee3558f14mpu4fq-ea-cluster", pod="ingress-operator-58b769b7cd-trz9l", severity="warning" Description: Pod ocm-arohcpci01-2sdigjqmlu1hialupee3558f14mpu4fq-ea-cluster/ingress-operator-58b769b7cd-trz9l has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-08-25T11:14:44Z Ended: 2026-08-25T11:21:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihks11c1ah6qljd4mi7jgu0gu4n2v-ea-listfdq7jq", pod="etcd-0", severity="warning" Description: Pod ocm-arohcpci01-2sdihks11c1ah6qljd4mi7jgu0gu4n2v-ea-listfdq7jq/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 5: State: Resolved Started: 2026-08-25T11:14:44Z Ended: 2026-08-25T11:20:44Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihsu90nkfp7klvqiibrq8qignpkn3-g2n5z8y2z6s4g5l", pod="etcd-1", severity="warning" Description: Pod ocm-arohcpci01-2sdihsu90nkfp7klvqiibrq8qignpkn3-g2n5z8y2z6s4g5l/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 6: State: Resolved Started: 2026-08-25T11:14:44Z Ended: 2026-08-25T11:21:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigjsm8auo5mc15flc0h8lullatcjv-t5i3m0e7c7g2q4m", pod="etcd-1", severity="warning" Description: Pod ocm-arohcpci01-2sdigjsm8auo5mc15flc0h8lullatcjv-t5i3m0e7c7g2q4m/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 7: State: Resolved Started: 2026-08-25T11:14:44Z Ended: 2026-08-25T11:26:46Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigjqmlu1hialupee3558f14mpu4fq-ea-cluster", pod="cluster-network-operator-55979955d9-chts5", severity="warning" Description: Pod ocm-arohcpci01-2sdigjqmlu1hialupee3558f14mpu4fq-ea-cluster/cluster-network-operator-55979955d9-chts5 has been in a non-ready state for longer than 5 minutes. Firing 8: State: Resolved Started: 2026-08-25T11:14:44Z Ended: 2026-08-25T11:21:44Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdih0rv1416jugfpm4c1sbi7ksv10ql-b1f1n2v7g9u5e7b", pod="etcd-2", severity="warning" Description: Pod ocm-arohcpci01-2sdih0rv1416jugfpm4c1sbi7ksv10ql-b1f1n2v7g9u5e7b/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 9: State: Resolved Started: 2026-08-25T11:14:44Z Ended: 2026-08-25T11:21:46Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigjqmlu1hialupee3558f14mpu4fq-ea-cluster", pod="cluster-policy-controller-b98bc7bb9-khp8v", severity="warning" Description: Pod ocm-arohcpci01-2sdigjqmlu1hialupee3558f14mpu4fq-ea-cluster/cluster-policy-controller-b98bc7bb9-khp8v has been in a non-ready state for longer than 5 minutes. Firing 10: State: Resolved Started: 2026-08-25T11:17:44Z Ended: 2026-08-25T11:24:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihcijeq1audlkvcatvpfdpj8kma3b-j9a9x3a9c6m6m9l", pod="etcd-0", severity="warning" Description: Pod ocm-arohcpci01-2sdihcijeq1audlkvcatvpfdpj8kma3b-j9a9x3a9c6m6m9l/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 11: State: Resolved Started: 2026-08-25T11:17:44Z Ended: 2026-08-25T11:24:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihkgjeab02r8c0v0399qpv5f7jdpv-private-ingress", pod="etcd-2", severity="warning" Description: Pod ocm-arohcpci01-2sdihkgjeab02r8c0v0399qpv5f7jdpv-private-ingress/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 12: State: Resolved Started: 2026-08-25T11:17:44Z Ended: 2026-08-25T11:26:44Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihv2u02bgedle3t00he6napcr1rvi-agg-cluster", pod="router-86db9b6b78-2d4wm", severity="warning" Description: Pod ocm-arohcpci01-2sdihv2u02bgedle3t00he6napcr1rvi-agg-cluster/router-86db9b6b78-2d4wm has been in a non-ready state for longer than 5 minutes. Firing 13: State: Resolved Started: 2026-08-25T11:17:44Z Ended: 2026-08-25T11:30:46Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihcijeq1audlkvcatvpfdpj8kma3b-j9a9x3a9c6m6m9l", pod="router-77bc884b98-2lq4f", severity="warning" Description: Pod ocm-arohcpci01-2sdihcijeq1audlkvcatvpfdpj8kma3b-j9a9x3a9c6m6m9l/router-77bc884b98-2lq4f has been in a non-ready state for longer than 5 minutes. Firing 14: State: Resolved Started: 2026-08-25T11:17:44Z Ended: 2026-08-25T11:24:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihcijeq1audlkvcatvpfdpj8kma3b-j9a9x3a9c6m6m9l", pod="etcd-1", severity="warning" Description: Pod ocm-arohcpci01-2sdihcijeq1audlkvcatvpfdpj8kma3b-j9a9x3a9c6m6m9l/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 15: State: Resolved Started: 2026-08-25T11:18:42Z Ended: 2026-08-25T11:24:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdih0u0tgeba2q4ut4gdbavl2edkv0e-n9p2o8n5f5n4v3c", pod="csi-snapshot-controller-operator-5b78c5479b-vxvgz", severity="warning" Description: Pod ocm-arohcpci01-2sdih0u0tgeba2q4ut4gdbavl2edkv0e-n9p2o8n5f5n4v3c/csi-snapshot-controller-operator-5b78c5479b-vxvgz has been in a non-ready state for longer than 5 minutes. Firing 16: State: Resolved Started: 2026-08-25T11:18:42Z Ended: 2026-08-25T11:24:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdih0u0tgeba2q4ut4gdbavl2edkv0e-n9p2o8n5f5n4v3c", pod="dns-operator-7d8cffff6-4qzgx", severity="warning" Description: Pod ocm-arohcpci01-2sdih0u0tgeba2q4ut4gdbavl2edkv0e-n9p2o8n5f5n4v3c/dns-operator-7d8cffff6-4qzgx has been in a non-ready state for longer than 5 minutes. Firing 17: State: Resolved Started: 2026-08-25T11:18:42Z Ended: 2026-08-25T11:24:44Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihhuv0ee4hrgbaa3e8v6e4l0g5vp2-i6a9q6l3u0w2l6i", pod="cluster-image-registry-operator-9cd45fd9c-rmjdj", severity="warning" Description: Pod ocm-arohcpci01-2sdihhuv0ee4hrgbaa3e8v6e4l0g5vp2-i6a9q6l3u0w2l6i/cluster-image-registry-operator-9cd45fd9c-rmjdj has been in a non-ready state for longer than 5 minutes. Firing 18: State: Resolved Started: 2026-08-25T11:18:43Z Ended: 2026-08-25T11:24:44Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihftlhoa7up08rja7n6ld8cgfrvpa-e3s2m5z1z4m6v2v", pod="openshift-apiserver-c6b8f956c-k45fh", severity="warning" Description: Pod ocm-arohcpci01-2sdihftlhoa7up08rja7n6ld8cgfrvpa-e3s2m5z1z4m6v2v/openshift-apiserver-c6b8f956c-k45fh has been in a non-ready state for longer than 5 minutes. Firing 19: State: Resolved Started: 2026-08-25T11:18:43Z Ended: 2026-08-25T11:27:47Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdih0u0tgeba2q4ut4gdbavl2edkv0e-n9p2o8n5f5n4v3c", pod="cluster-network-operator-65ccfd54c7-8x6vv", severity="warning" Description: Pod ocm-arohcpci01-2sdih0u0tgeba2q4ut4gdbavl2edkv0e-n9p2o8n5f5n4v3c/cluster-network-operator-65ccfd54c7-8x6vv has been in a non-ready state for longer than 5 minutes. Firing 20: State: Resolved Started: 2026-08-25T11:18:43Z Ended: 2026-08-25T11:24:44Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihftlhoa7up08rja7n6ld8cgfrvpa-e3s2m5z1z4m6v2v", pod="ignition-server-proxy-5c4786958-7fw5g", severity="warning" Description: Pod ocm-arohcpci01-2sdihftlhoa7up08rja7n6ld8cgfrvpa-e3s2m5z1z4m6v2v/ignition-server-proxy-5c4786958-7fw5g has been in a non-ready state for longer than 5 minutes. Firing 21: State: Resolved Started: 2026-08-25T11:18:44Z Ended: 2026-08-25T11:24:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihhuv0ee4hrgbaa3e8v6e4l0g5vp2-i6a9q6l3u0w2l6i", pod="csi-snapshot-controller-operator-786784fd7b-vnl4v", severity="warning" Description: Pod ocm-arohcpci01-2sdihhuv0ee4hrgbaa3e8v6e4l0g5vp2-i6a9q6l3u0w2l6i/csi-snapshot-controller-operator-786784fd7b-vnl4v has been in a non-ready state for longer than 5 minutes. Firing 22: State: Resolved Started: 2026-08-25T11:18:44Z Ended: 2026-08-25T11:24:44Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdih0u0tgeba2q4ut4gdbavl2edkv0e-n9p2o8n5f5n4v3c", pod="cluster-storage-operator-78f86b5d97-r2zsk", severity="warning" Description: Pod ocm-arohcpci01-2sdih0u0tgeba2q4ut4gdbavl2edkv0e-n9p2o8n5f5n4v3c/cluster-storage-operator-78f86b5d97-r2zsk has been in a non-ready state for longer than 5 minutes. Firing 23: State: Resolved Started: 2026-08-25T11:18:44Z Ended: 2026-08-25T11:28:42Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdih0u0tgeba2q4ut4gdbavl2edkv0e-n9p2o8n5f5n4v3c", pod="ingress-operator-6c97f54695-s68pf", severity="warning" Description: Pod ocm-arohcpci01-2sdih0u0tgeba2q4ut4gdbavl2edkv0e-n9p2o8n5f5n4v3c/ingress-operator-6c97f54695-s68pf has been in a non-ready state for longer than 5 minutes. Firing 24: State: Resolved Started: 2026-08-25T11:18:44Z Ended: 2026-08-25T11:24:46Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdih0u0tgeba2q4ut4gdbavl2edkv0e-n9p2o8n5f5n4v3c", pod="cluster-version-operator-67d94cd64d-kbh9b", severity="warning" Description: Pod ocm-arohcpci01-2sdih0u0tgeba2q4ut4gdbavl2edkv0e-n9p2o8n5f5n4v3c/cluster-version-operator-67d94cd64d-kbh9b has been in a non-ready state for longer than 5 minutes. Firing 25: State: Resolved Started: 2026-08-25T11:18:44Z Ended: 2026-08-25T11:24:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihhuv0ee4hrgbaa3e8v6e4l0g5vp2-i6a9q6l3u0w2l6i", pod="cluster-policy-controller-749c586f99-nwcsd", severity="warning" Description: Pod ocm-arohcpci01-2sdihhuv0ee4hrgbaa3e8v6e4l0g5vp2-i6a9q6l3u0w2l6i/cluster-policy-controller-749c586f99-nwcsd has been in a non-ready state for longer than 5 minutes. Firing 26: State: Resolved Started: 2026-08-25T11:19:42Z Ended: 2026-08-25T11:27:48Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigrgeahe87bis0ukc0jhq92p044to-b9f1c2q0b1l8t4p", pod="cluster-network-operator-5d4f6bdfbb-8qzx6", severity="warning" Description: Pod ocm-arohcpci01-2sdigrgeahe87bis0ukc0jhq92p044to-b9f1c2q0b1l8t4p/cluster-network-operator-5d4f6bdfbb-8qzx6 has been in a non-ready state for longer than 5 minutes. Firing 27: State: Resolved Started: 2026-08-25T11:19:42Z Ended: 2026-08-25T11:25:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihknfg29t9kvecdq95udbscb5i26m-cilium-cluster", pod="router-7fd5474f96-mgmpb", severity="warning" Description: Pod ocm-arohcpci01-2sdihknfg29t9kvecdq95udbscb5i26m-cilium-cluster/router-7fd5474f96-mgmpb has been in a non-ready state for longer than 5 minutes. Firing 28: State: Resolved Started: 2026-08-25T11:19:42Z Ended: 2026-08-25T11:29:44Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigrgeahe87bis0ukc0jhq92p044to-b9f1c2q0b1l8t4p", pod="dns-operator-ff7889f8d-5sphm", severity="warning" Description: Pod ocm-arohcpci01-2sdigrgeahe87bis0ukc0jhq92p044to-b9f1c2q0b1l8t4p/dns-operator-ff7889f8d-5sphm has been in a non-ready state for longer than 5 minutes. Firing 29: State: Resolved Started: 2026-08-25T11:19:42Z Ended: 2026-08-25T11:25:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigme1iva31aldq1cv323josrfntfm-o9z2z2j3o7m8n4t", pod="openshift-apiserver-6575b9fb56-9crm7", severity="warning" Description: Pod ocm-arohcpci01-2sdigme1iva31aldq1cv323josrfntfm-o9z2z2j3o7m8n4t/openshift-apiserver-6575b9fb56-9crm7 has been in a non-ready state for longer than 5 minutes. Firing 30: State: Resolved Started: 2026-08-25T11:19:42Z Ended: 2026-08-25T11:26:44Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihv2u02bgedle3t00he6napcr1rvi-agg-cluster", pod="kube-apiserver-55f4db9d5-jrvtz", severity="warning" Description: Pod ocm-arohcpci01-2sdihv2u02bgedle3t00he6napcr1rvi-agg-cluster/kube-apiserver-55f4db9d5-jrvtz has been in a non-ready state for longer than 5 minutes. Firing 31: State: Resolved Started: 2026-08-25T11:19:42Z Ended: 2026-08-25T11:25:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigov251107me410shcf9af2facoqc-oidc-wi-cluster", pod="openshift-apiserver-6f6d6b88f9-b5zrt", severity="warning" Description: Pod ocm-arohcpci01-2sdigov251107me410shcf9af2facoqc-oidc-wi-cluster/openshift-apiserver-6f6d6b88f9-b5zrt has been in a non-ready state for longer than 5 minutes. Firing 32: State: Resolved Started: 2026-08-25T11:19:43Z Ended: 2026-08-25T11:27:47Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigrgeahe87bis0ukc0jhq92p044to-b9f1c2q0b1l8t4p", pod="ingress-operator-5c7c8cb448-8cqf4", severity="warning" Description: Pod ocm-arohcpci01-2sdigrgeahe87bis0ukc0jhq92p044to-b9f1c2q0b1l8t4p/ingress-operator-5c7c8cb448-8cqf4 has been in a non-ready state for longer than 5 minutes. Firing 33: State: Resolved Started: 2026-08-25T11:19:43Z Ended: 2026-08-25T11:27:46Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigrgeahe87bis0ukc0jhq92p044to-b9f1c2q0b1l8t4p", pod="cluster-storage-operator-55f8b9d749-f6jpd", severity="warning" Description: Pod ocm-arohcpci01-2sdigrgeahe87bis0ukc0jhq92p044to-b9f1c2q0b1l8t4p/cluster-storage-operator-55f8b9d749-f6jpd has been in a non-ready state for longer than 5 minutes. Firing 34: State: Resolved Started: 2026-08-25T11:19:43Z Ended: 2026-08-25T11:27:46Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigrgeahe87bis0ukc0jhq92p044to-b9f1c2q0b1l8t4p", pod="csi-snapshot-controller-operator-b779dc57c-dvvv2", severity="warning" Description: Pod ocm-arohcpci01-2sdigrgeahe87bis0ukc0jhq92p044to-b9f1c2q0b1l8t4p/csi-snapshot-controller-operator-b779dc57c-dvvv2 has been in a non-ready state for longer than 5 minutes. Firing 35: State: Resolved Started: 2026-08-25T11:20:43Z Ended: 2026-08-25T11:31:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdih0fp1fmn6t6fkvkmpfvi3s509kvi-l3u6d4n8d9v0b0o", pod="router-849cb6d568-v29p5", severity="warning" Description: Pod ocm-arohcpci01-2sdih0fp1fmn6t6fkvkmpfvi3s509kvi-l3u6d4n8d9v0b0o/router-849cb6d568-v29p5 has been in a non-ready state for longer than 5 minutes. Firing 36: State: Resolved Started: 2026-08-25T11:20:43Z Ended: 2026-08-25T11:30:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihho1a123mh2it2n7vctui881hqae-w3e6w3f0k2r0k3f", pod="ingress-operator-869d954d45-zjgp9", severity="warning" Description: Pod ocm-arohcpci01-2sdihho1a123mh2it2n7vctui881hqae-w3e6w3f0k2r0k3f/ingress-operator-869d954d45-zjgp9 has been in a non-ready state for longer than 5 minutes. Firing 37: State: Resolved Started: 2026-08-25T11:20:44Z Ended: 2026-08-25T11:26:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihho1a123mh2it2n7vctui881hqae-w3e6w3f0k2r0k3f", pod="dns-operator-57f67c6dd6-5llhc", severity="warning" Description: Pod ocm-arohcpci01-2sdihho1a123mh2it2n7vctui881hqae-w3e6w3f0k2r0k3f/dns-operator-57f67c6dd6-5llhc has been in a non-ready state for longer than 5 minutes. Firing 38: State: Resolved Started: 2026-08-25T11:20:44Z Ended: 2026-08-25T11:30:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihho1a123mh2it2n7vctui881hqae-w3e6w3f0k2r0k3f", pod="cluster-network-operator-56d944cdf4-4jmzb", severity="warning" Description: Pod ocm-arohcpci01-2sdihho1a123mh2it2n7vctui881hqae-w3e6w3f0k2r0k3f/cluster-network-operator-56d944cdf4-4jmzb has been in a non-ready state for longer than 5 minutes. Firing 39: State: Resolved Started: 2026-08-25T11:21:43Z Ended: 2026-08-25T11:27:47Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigh8k98csmst57kjmi4q90sgs2rrv-p1f9b4x3p6l5s0l", pod="dns-operator-7bb8484758-wq6pf", severity="warning" Description: Pod ocm-arohcpci01-2sdigh8k98csmst57kjmi4q90sgs2rrv-p1f9b4x3p6l5s0l/dns-operator-7bb8484758-wq6pf has been in a non-ready state for longer than 5 minutes. Firing 40: State: Resolved Started: 2026-08-25T11:21:43Z Ended: 2026-08-25T11:27:48Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigh8k98csmst57kjmi4q90sgs2rrv-p1f9b4x3p6l5s0l", pod="ingress-operator-67bb49d9f8-txgcd", severity="warning" Description: Pod ocm-arohcpci01-2sdigh8k98csmst57kjmi4q90sgs2rrv-p1f9b4x3p6l5s0l/ingress-operator-67bb49d9f8-txgcd has been in a non-ready state for longer than 5 minutes. Firing 41: State: Resolved Started: 2026-08-25T11:21:43Z Ended: 2026-08-25T11:27:47Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdih2lpap41tb37m3d5e0kjrh3jjumd-v20260901", pod="cluster-network-operator-7cbf6895bf-9bh5v", severity="warning" Description: Pod ocm-arohcpci01-2sdih2lpap41tb37m3d5e0kjrh3jjumd-v20260901/cluster-network-operator-7cbf6895bf-9bh5v has been in a non-ready state for longer than 5 minutes. Firing 42: State: Resolved Started: 2026-08-25T11:21:43Z Ended: 2026-08-25T11:27:46Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihhvcgi49scqvaqim6cattb486hhm-a8p1s8n1b6h5h6p", pod="openshift-apiserver-65b67c685b-wsrvn", severity="warning" Description: Pod ocm-arohcpci01-2sdihhvcgi49scqvaqim6cattb486hhm-a8p1s8n1b6h5h6p/openshift-apiserver-65b67c685b-wsrvn has been in a non-ready state for longer than 5 minutes. Firing 43: State: Resolved Started: 2026-08-25T11:21:43Z Ended: 2026-08-25T11:29:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihhs26bbe32b2vccmrv2fet56h328-a1x4k9z1m7q2p5b", pod="ingress-operator-74795fb8c8-ks7sl", severity="warning" Description: Pod ocm-arohcpci01-2sdihhs26bbe32b2vccmrv2fet56h328-a1x4k9z1m7q2p5b/ingress-operator-74795fb8c8-ks7sl has been in a non-ready state for longer than 5 minutes. Firing 44: State: Resolved Started: 2026-08-25T11:21:43Z Ended: 2026-08-25T11:27:46Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihhso6vshstvo53lrpih930g1l92g-sre-hcp-cluster", pod="ingress-operator-7b7448bb88-6zb7c", severity="warning" Description: Pod ocm-arohcpci01-2sdihhso6vshstvo53lrpih930g1l92g-sre-hcp-cluster/ingress-operator-7b7448bb88-6zb7c has been in a non-ready state for longer than 5 minutes. Firing 45: State: Resolved Started: 2026-08-25T11:21:43Z Ended: 2026-08-25T11:27:46Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigh8k98csmst57kjmi4q90sgs2rrv-p1f9b4x3p6l5s0l", pod="dns-operator-f554cd46f-kznkx", severity="warning" Description: Pod ocm-arohcpci01-2sdigh8k98csmst57kjmi4q90sgs2rrv-p1f9b4x3p6l5s0l/dns-operator-f554cd46f-kznkx has been in a non-ready state for longer than 5 minutes. Firing 46: State: Resolved Started: 2026-08-25T11:21:44Z Ended: 2026-08-25T11:27:47Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigh8k98csmst57kjmi4q90sgs2rrv-p1f9b4x3p6l5s0l", pod="cluster-network-operator-69749475d4-ksxdv", severity="warning" Description: Pod ocm-arohcpci01-2sdigh8k98csmst57kjmi4q90sgs2rrv-p1f9b4x3p6l5s0l/cluster-network-operator-69749475d4-ksxdv has been in a non-ready state for longer than 5 minutes. Firing 47: State: Resolved Started: 2026-08-25T11:21:44Z Ended: 2026-08-25T11:27:47Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigoilh0gkbc6gnpu9m8c8j992l9qo-g4f2e7b8q7i0m5c", pod="kube-apiserver-7bf5897fbd-llw26", severity="warning" Description: Pod ocm-arohcpci01-2sdigoilh0gkbc6gnpu9m8c8j992l9qo-g4f2e7b8q7i0m5c/kube-apiserver-7bf5897fbd-llw26 has been in a non-ready state for longer than 5 minutes. Firing 48: State: Resolved Started: 2026-08-25T11:21:44Z Ended: 2026-08-25T11:27:48Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdih2lpap41tb37m3d5e0kjrh3jjumd-v20260901", pod="ingress-operator-5984c78bc-qprwn", severity="warning" Description: Pod ocm-arohcpci01-2sdih2lpap41tb37m3d5e0kjrh3jjumd-v20260901/ingress-operator-5984c78bc-qprwn has been in a non-ready state for longer than 5 minutes. Firing 49: State: Resolved Started: 2026-08-25T11:21:44Z Ended: 2026-08-25T11:29:44Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdih2lpap41tb37m3d5e0kjrh3jjumd-v20260901", pod="dns-operator-5c9cd96b99-t8sp9", severity="warning" Description: Pod ocm-arohcpci01-2sdih2lpap41tb37m3d5e0kjrh3jjumd-v20260901/dns-operator-5c9cd96b99-t8sp9 has been in a non-ready state for longer than 5 minutes. Firing 50: State: Resolved Started: 2026-08-25T11:21:44Z Ended: 2026-08-25T11:29:46Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigh8k98csmst57kjmi4q90sgs2rrv-p1f9b4x3p6l5s0l", pod="cluster-image-registry-operator-799f4846c9-hsr8m", severity="warning" Description: Pod ocm-arohcpci01-2sdigh8k98csmst57kjmi4q90sgs2rrv-p1f9b4x3p6l5s0l/cluster-image-registry-operator-799f4846c9-hsr8m has been in a non-ready state for longer than 5 minutes. Firing 51: State: Resolved Started: 2026-08-25T11:22:45Z Ended: 2026-08-25T11:28:42Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihifj6r0cdpo4eenhniqoepqkarqg-d1l0y1w8p6m0t0v", pod="ingress-operator-5dbdc46f56-fc76w", severity="warning" Description: Pod ocm-arohcpci01-2sdihifj6r0cdpo4eenhniqoepqkarqg-d1l0y1w8p6m0t0v/ingress-operator-5dbdc46f56-fc76w has been in a non-ready state for longer than 5 minutes. Firing 52: State: Resolved Started: 2026-08-25T11:22:45Z Ended: 2026-08-25T11:29:47Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihhutt6egf8nkbaeanni91sjqjthb-w0v1j4o3n3d9j5t", pod="ingress-operator-66d96f9d4c-42j9m", severity="warning" Description: Pod ocm-arohcpci01-2sdihhutt6egf8nkbaeanni91sjqjthb-w0v1j4o3n3d9j5t/ingress-operator-66d96f9d4c-42j9m has been in a non-ready state for longer than 5 minutes. Firing 53: State: Resolved Started: 2026-08-25T11:22:45Z Ended: 2026-08-25T11:31:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihhvcgi49scqvaqim6cattb486hhm-a8p1s8n1b6h5h6p", pod="router-c456f5fbc-cpx28", severity="warning" Description: Pod ocm-arohcpci01-2sdihhvcgi49scqvaqim6cattb486hhm-a8p1s8n1b6h5h6p/router-c456f5fbc-cpx28 has been in a non-ready state for longer than 5 minutes. Firing 54: State: Resolved Started: 2026-08-25T11:23:44Z Ended: 2026-08-25T11:29:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihfmqfe5p85hkrqrs8mjlc5eja6jc-i2i1n6g2o3d6n5s", pod="dns-operator-56478c8694-r5thf", severity="warning" Description: Pod ocm-arohcpci01-2sdihfmqfe5p85hkrqrs8mjlc5eja6jc-i2i1n6g2o3d6n5s/dns-operator-56478c8694-r5thf has been in a non-ready state for longer than 5 minutes. Firing 55: State: Resolved Started: 2026-08-25T11:23:44Z Ended: 2026-08-25T11:29:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihfmqfe5p85hkrqrs8mjlc5eja6jc-i2i1n6g2o3d6n5s", pod="csi-snapshot-controller-operator-85d4568778-zpnvq", severity="warning" Description: Pod ocm-arohcpci01-2sdihfmqfe5p85hkrqrs8mjlc5eja6jc-i2i1n6g2o3d6n5s/csi-snapshot-controller-operator-85d4568778-zpnvq has been in a non-ready state for longer than 5 minutes. Firing 56: State: Resolved Started: 2026-08-25T11:23:44Z Ended: 2026-08-25T11:29:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdighetrfta0j8u1pa03dtl5k9q9d0n-a8c8t3q7i8y6n4z", pod="openshift-route-controller-manager-548849d46-7ksmw", severity="warning" Description: Pod ocm-arohcpci01-2sdighetrfta0j8u1pa03dtl5k9q9d0n-a8c8t3q7i8y6n4z/openshift-route-controller-manager-548849d46-7ksmw has been in a non-ready state for longer than 5 minutes. Firing 57: State: Resolved Started: 2026-08-25T11:23:44Z Ended: 2026-08-25T11:29:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihhutt6egf8nkbaeanni91sjqjthb-w0v1j4o3n3d9j5t", pod="azure-disk-csi-driver-operator-69b864d46c-nsqcn", severity="warning" Description: Pod ocm-arohcpci01-2sdihhutt6egf8nkbaeanni91sjqjthb-w0v1j4o3n3d9j5t/azure-disk-csi-driver-operator-69b864d46c-nsqcn has been in a non-ready state for longer than 5 minutes. Firing 58: State: Resolved Started: 2026-08-25T11:23:44Z Ended: 2026-08-25T11:29:47Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigjtvr1u96r8k47ihdtfqsispp6gk-ephemeral-disk", pod="network-node-identity-7f66fcbc7-jl89z", severity="warning" Description: Pod ocm-arohcpci01-2sdigjtvr1u96r8k47ihdtfqsispp6gk-ephemeral-disk/network-node-identity-7f66fcbc7-jl89z has been in a non-ready state for longer than 5 minutes. Firing 59: State: Resolved Started: 2026-08-25T11:23:44Z Ended: 2026-08-25T11:29:46Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihi1m0v5mh13qjfkn5df7i1q7fivq-l5t4f9d3i5i4l3v", pod="azure-cloud-controller-manager-77fdc894b8-b6lwr", severity="warning" Description: Pod ocm-arohcpci01-2sdihi1m0v5mh13qjfkn5df7i1q7fivq-l5t4f9d3i5i4l3v/azure-cloud-controller-manager-77fdc894b8-b6lwr has been in a non-ready state for longer than 5 minutes. Firing 60: State: Resolved Started: 2026-08-25T11:23:44Z Ended: 2026-08-25T11:29:46Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihfmqfe5p85hkrqrs8mjlc5eja6jc-i2i1n6g2o3d6n5s", pod="cluster-version-operator-65df5db46b-tlhsl", severity="warning" Description: Pod ocm-arohcpci01-2sdihfmqfe5p85hkrqrs8mjlc5eja6jc-i2i1n6g2o3d6n5s/cluster-version-operator-65df5db46b-tlhsl has been in a non-ready state for longer than 5 minutes. Firing 61: State: Resolved Started: 2026-08-25T11:23:44Z Ended: 2026-08-25T11:29:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihfmqfe5p85hkrqrs8mjlc5eja6jc-i2i1n6g2o3d6n5s", pod="cluster-storage-operator-567d9c98b5-6sdl4", severity="warning" Description: Pod ocm-arohcpci01-2sdihfmqfe5p85hkrqrs8mjlc5eja6jc-i2i1n6g2o3d6n5s/cluster-storage-operator-567d9c98b5-6sdl4 has been in a non-ready state for longer than 5 minutes. Firing 62: State: Resolved Started: 2026-08-25T11:23:44Z Ended: 2026-08-25T11:29:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihfmqfe5p85hkrqrs8mjlc5eja6jc-i2i1n6g2o3d6n5s", pod="dns-operator-7647df8fc9-vx29n", severity="warning" Description: Pod ocm-arohcpci01-2sdihfmqfe5p85hkrqrs8mjlc5eja6jc-i2i1n6g2o3d6n5s/dns-operator-7647df8fc9-vx29n has been in a non-ready state for longer than 5 minutes. Firing 63: State: Resolved Started: 2026-08-25T11:23:44Z Ended: 2026-08-25T11:29:46Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihht90lpuf6ep0e3v2mivrdmdj41b-cilium-cl", pod="network-node-identity-5cd8ddcdc-5xhcd", severity="warning" Description: Pod ocm-arohcpci01-2sdihht90lpuf6ep0e3v2mivrdmdj41b-cilium-cl/network-node-identity-5cd8ddcdc-5xhcd has been in a non-ready state for longer than 5 minutes. Firing 64: State: Resolved Started: 2026-08-25T11:23:44Z Ended: 2026-08-25T11:29:46Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdig96jq9lrl1uj7d4bbvn1vv0tvn7n-i8z5u7n3r5s9m7x", pod="openshift-controller-manager-7fc7cf7599-xdwld", severity="warning" Description: Pod ocm-arohcpci01-2sdig96jq9lrl1uj7d4bbvn1vv0tvn7n-i8z5u7n3r5s9m7x/openshift-controller-manager-7fc7cf7599-xdwld has been in a non-ready state for longer than 5 minutes. Firing 65: State: Resolved Started: 2026-08-25T11:23:44Z Ended: 2026-08-25T11:29:46Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihks11c1ah6qljd4mi7jgu0gu4n2v-ea-listfdq7jq", pod="openshift-apiserver-6b65f797cd-t68bj", severity="warning" Description: Pod ocm-arohcpci01-2sdihks11c1ah6qljd4mi7jgu0gu4n2v-ea-listfdq7jq/openshift-apiserver-6b65f797cd-t68bj has been in a non-ready state for longer than 5 minutes. Firing 66: State: Resolved Started: 2026-08-25T11:23:44Z Ended: 2026-08-25T11:29:47Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihhsl4qrml5mn6tj1fociu37kug12-l9m0t9e6f1g2x0z", pod="openshift-apiserver-759c44dc9f-n2pwg", severity="warning" Description: Pod ocm-arohcpci01-2sdihhsl4qrml5mn6tj1fociu37kug12-l9m0t9e6f1g2x0z/openshift-apiserver-759c44dc9f-n2pwg has been in a non-ready state for longer than 5 minutes. Firing 67: State: Resolved Started: 2026-08-25T11:23:44Z Ended: 2026-08-25T11:29:46Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdighetrfta0j8u1pa03dtl5k9q9d0n-a8c8t3q7i8y6n4z", pod="openshift-controller-manager-9854d9b75-hlthh", severity="warning" Description: Pod ocm-arohcpci01-2sdighetrfta0j8u1pa03dtl5k9q9d0n-a8c8t3q7i8y6n4z/openshift-controller-manager-9854d9b75-hlthh has been in a non-ready state for longer than 5 minutes. Firing 68: State: Resolved Started: 2026-08-25T11:23:44Z Ended: 2026-08-25T11:32:42Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihfmqfe5p85hkrqrs8mjlc5eja6jc-i2i1n6g2o3d6n5s", pod="cluster-policy-controller-649687f75f-7pxlr", severity="warning" Description: Pod ocm-arohcpci01-2sdihfmqfe5p85hkrqrs8mjlc5eja6jc-i2i1n6g2o3d6n5s/cluster-policy-controller-649687f75f-7pxlr has been in a non-ready state for longer than 5 minutes. Firing 69: State: Resolved Started: 2026-08-25T11:23:44Z Ended: 2026-08-25T11:29:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihfmqfe5p85hkrqrs8mjlc5eja6jc-i2i1n6g2o3d6n5s", pod="ingress-operator-6545876586-9w5gg", severity="warning" Description: Pod ocm-arohcpci01-2sdihfmqfe5p85hkrqrs8mjlc5eja6jc-i2i1n6g2o3d6n5s/ingress-operator-6545876586-9w5gg has been in a non-ready state for longer than 5 minutes. Firing 70: State: Resolved Started: 2026-08-25T11:23:44Z Ended: 2026-08-25T11:29:47Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdiht06nip65lnjsqasans87pbijccq-r3e9n8e0v6j4q6w", pod="cluster-policy-controller-6c4d44dc98-frm2c", severity="warning" Description: Pod ocm-arohcpci01-2sdiht06nip65lnjsqasans87pbijccq-r3e9n8e0v6j4q6w/cluster-policy-controller-6c4d44dc98-frm2c has been in a non-ready state for longer than 5 minutes. Firing 71: State: Resolved Started: 2026-08-25T11:23:44Z Ended: 2026-08-25T11:32:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihfmqfe5p85hkrqrs8mjlc5eja6jc-i2i1n6g2o3d6n5s", pod="cluster-network-operator-85f89f8fc4-4sq6t", severity="warning" Description: Pod ocm-arohcpci01-2sdihfmqfe5p85hkrqrs8mjlc5eja6jc-i2i1n6g2o3d6n5s/cluster-network-operator-85f89f8fc4-4sq6t has been in a non-ready state for longer than 5 minutes. Firing 72: State: Resolved Started: 2026-08-25T11:23:44Z Ended: 2026-08-25T11:29:46Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigos14te0s7edb7ob5g0vc9ploiqq-c2b0y6a1g3p3d2h", pod="azure-file-csi-driver-operator-bdc6cd8fb-vkxwm", severity="warning" Description: Pod ocm-arohcpci01-2sdigos14te0s7edb7ob5g0vc9ploiqq-c2b0y6a1g3p3d2h/azure-file-csi-driver-operator-bdc6cd8fb-vkxwm has been in a non-ready state for longer than 5 minutes. Firing 73: State: Resolved Started: 2026-08-25T11:23:44Z Ended: 2026-08-25T11:29:46Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihv4p7b4sh7keln37oq2g0um64blt-private-kas", pod="kube-apiserver-6fb9877758-ftskx", severity="warning" Description: Pod ocm-arohcpci01-2sdihv4p7b4sh7keln37oq2g0um64blt-private-kas/kube-apiserver-6fb9877758-ftskx has been in a non-ready state for longer than 5 minutes. Firing 74: State: Resolved Started: 2026-08-25T11:23:44Z Ended: 2026-08-25T11:32:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihhuv0ee4hrgbaa3e8v6e4l0g5vp2-i6a9q6l3u0w2l6i", pod="azure-disk-csi-driver-controller-5f8c6c7c87-m4x62", severity="warning" Description: Pod ocm-arohcpci01-2sdihhuv0ee4hrgbaa3e8v6e4l0g5vp2-i6a9q6l3u0w2l6i/azure-disk-csi-driver-controller-5f8c6c7c87-m4x62 has been in a non-ready state for longer than 5 minutes. Firing 75: State: Resolved Started: 2026-08-25T11:24:42Z Ended: 2026-08-25T11:32:42Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdighetrfta0j8u1pa03dtl5k9q9d0n-a8c8t3q7i8y6n4z", pod="multus-admission-controller-55c5df6f5f-xkh8d", severity="warning" Description: Pod ocm-arohcpci01-2sdighetrfta0j8u1pa03dtl5k9q9d0n-a8c8t3q7i8y6n4z/multus-admission-controller-55c5df6f5f-xkh8d has been in a non-ready state for longer than 5 minutes. Firing 76: State: Resolved Started: 2026-08-25T11:24:42Z Ended: 2026-08-25T11:30:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihi9v807i18d9cojts3mvodckr2de-f9a5n9z1j4t3j2e", pod="ingress-operator-547bc47776-h9jrq", severity="warning" Description: Pod ocm-arohcpci01-2sdihi9v807i18d9cojts3mvodckr2de-f9a5n9z1j4t3j2e/ingress-operator-547bc47776-h9jrq has been in a non-ready state for longer than 5 minutes. Firing 77: State: Resolved Started: 2026-08-25T11:24:42Z Ended: 2026-08-25T11:30:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihi1m9ldavl96j49nv4088qcmig2f-t6m4j9a7p4l7y2n", pod="cluster-network-operator-7c7bcfc67c-qm7mw", severity="warning" Description: Pod ocm-arohcpci01-2sdihi1m9ldavl96j49nv4088qcmig2f-t6m4j9a7p4l7y2n/cluster-network-operator-7c7bcfc67c-qm7mw has been in a non-ready state for longer than 5 minutes. Firing 78: State: Resolved Started: 2026-08-25T11:24:43Z Ended: 2026-08-25T11:30:46Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihi9v807i18d9cojts3mvodckr2de-f9a5n9z1j4t3j2e", pod="cluster-image-registry-operator-dc7fbc54c-ntqk2", severity="warning" Description: Pod ocm-arohcpci01-2sdihi9v807i18d9cojts3mvodckr2de-f9a5n9z1j4t3j2e/cluster-image-registry-operator-dc7fbc54c-ntqk2 has been in a non-ready state for longer than 5 minutes. Firing 79: State: Resolved Started: 2026-08-25T11:24:43Z Ended: 2026-08-25T11:30:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihi1m9ldavl96j49nv4088qcmig2f-t6m4j9a7p4l7y2n", pod="ingress-operator-6c84f87648-5hfq2", severity="warning" Description: Pod ocm-arohcpci01-2sdihi1m9ldavl96j49nv4088qcmig2f-t6m4j9a7p4l7y2n/ingress-operator-6c84f87648-5hfq2 has been in a non-ready state for longer than 5 minutes. Firing 80: State: Resolved Started: 2026-08-25T11:24:45Z Ended: 2026-08-25T11:30:46Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdigohasqldgia8f7o0a0h5i1fpok8s-m7a3r7h3q4f4n3a", pod="kube-storage-version-migrator-56b69bc9c8-5xctr", severity="warning" Description: Pod ocm-arohcpci01-2sdigohasqldgia8f7o0a0h5i1fpok8s-m7a3r7h3q4f4n3a/kube-storage-version-migrator-56b69bc9c8-5xctr has been in a non-ready state for longer than 5 minutes. Firing 81: State: Resolved Started: 2026-08-25T11:25:42Z Ended: 2026-08-25T11:32:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5488384-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdihknfg29t9kvecdq95udbscb5i26m-cilium-cluster", pod="router-68768c77b5-4h4f8", severity="warning" Description: Pod ocm-arohcpci01-2sdihknfg29t9kvecdq95udbscb5i26m-cilium-cluster/router-68768c77b5-4h4f8 has been in a non-ready state for longer than 5 minutes. alert [svc] UJClusterProvisionErrors6h30m fired 10:02:45 UTC · failed at alert · test [aro-hcp-observability] [svc] alert UJClusterProvisionErrors6h30m does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-08-25T11:52:21Z Severity: Sev3 Labels: alertname="UJClusterProvisionErrors6h30m", cluster="ci01-j5488384-svc", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5488384/providers/microsoft.monitor/accounts/services-j5488384", severity="3", short_window="30m", slo="cluster-provision-errors" Description: More than 30% of cluster create (install) operations are in failed state sustained over 30 minutes, indicating a medium error budget burn (6x) that would exhaust the 95% SLO budget in ~28 hours.
10:02:43	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c9b8452	e2e	failed	⚠#6584 (open)	1	e2e (1) ERROR CODE: ResourceNotFound; detail message The resource 'locations/hcpOperationStatuses/<uuid>' under resource group '' was not found.; provider Microsoft.RedHatOpenShift 10:02:43 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:217]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-lpd7ls" to 4.21 Unexpected error: <fmt.wrapError \| 0xc000a5c5a0>: failed waiting for hcpCluster="cp-ystream-upgrade-4-21-lpd7ls" in resourcegroup="rg-cp-ystream-upgrade-4-21-nkpr7nplpjkn" to finish updating: GET https://rp.j8801025.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a615d984-573b-4196-87e0-68aeb8a371a3 -------------------------------------------------------------------------------- RESPONSE 404: 404 Not Found ERROR CODE: ResourceNotFound -------------------------------------------------------------------------------- { "error": { "code": "ResourceNotFound", "message": "The resource 'locations/hcpOperationStatuses/a615d984-573b-4196-87e0-68aeb8a371a3' under resource group '' was not found.", "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a615d984-573b-4196-87e0-68aeb8a371a3" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:217]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-lpd7ls" to 4.21 Unexpected error: <fmt.wrapError \| 0xc000a5c5a0>: failed waiting for hcpCluster="cp-ystream-upgrade-4-21-lpd7ls" in resourcegroup="rg-cp-ystream-upgrade-4-21-nkpr7nplpjkn" to finish updating: GET https://rp.j8801025.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a615d984-573b-4196-87e0-68aeb8a371a3 -------------------------------------------------------------------------------- RESPONSE 404: 404 Not Found ERROR CODE: ResourceNotFound -------------------------------------------------------------------------------- { "error": { "code": "ResourceNotFound", "message": "The resource 'locations/hcpOperationStatuses/a615d984-573b-4196-87e0-68aeb8a371a3' under resource group '' was not found.", "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a615d984-573b-4196-87e0-68aeb8a371a3" } } -------------------------------------------------------------------------------- ... occurred
09:56:13	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 2cc8be4	unknown	failed	#6670 (open)	1	non-artifact-backed failure (no junit artifacts)
09:52:51	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 5c0ce63	e2e	failed	#6669 (open)	2	e2e (2) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift 09:52:51 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc0026c6318>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-krw8vkxmtzpb" to finish deleting: GET https://rp.j5592064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/74596e16-42cf-45f7-b5d0-faa23e1127f5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/74596e16-42cf-45f7-b5d0-faa23e1127f5", "name": "74596e16-42cf-45f7-b5d0-faa23e1127f5", "status": "Failed", "startTime": "2026-08-25T10:56:49.253326502Z", "endTime": "2026-08-25T11:20:52.069849414Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sdi5csg0pn2q641bbc4hk7ubn3ksvmp still exists (deletion dispatched at 2026-08-25T10:56:49Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc0026c6318>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-krw8vkxmtzpb" to finish deleting: GET https://rp.j5592064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/74596e16-42cf-45f7-b5d0-faa23e1127f5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/74596e16-42cf-45f7-b5d0-faa23e1127f5", "name": "74596e16-42cf-45f7-b5d0-faa23e1127f5", "status": "Failed", "startTime": "2026-08-25T10:56:49.253326502Z", "endTime": "2026-08-25T11:20:52.069849414Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sdi5csg0pn2q641bbc4hk7ubn3ksvmp still exists (deletion dispatched at 2026-08-25T10:56:49Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred failed to get latest install version for candidate channel: failed getting controlPlaneVersion: Get "<url> 09:52:51 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:143]: failed to get latest install version for candidate channel: failed getting controlPlaneVersion: Get "https://api.openshift.com/api/upgrades_info/graph?arch=multi&channel=candidate-4.20": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.129.200.17:33911->172.30.0.10:53: i/o timeout fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:143]: failed to get latest install version for candidate channel: failed getting controlPlaneVersion: Get "https://api.openshift.com/api/upgrades_info/graph?arch=multi&channel=candidate-4.20": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.129.200.17:33911->172.30.0.10:53: i/o timeout
09:47:22	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 2cc8be4	unknown	failed	#6670 (open)	1	non-artifact-backed failure (no junit artifacts)
09:45:00	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 1b73f10	unknown	failed	#6657 (open)	1	non-artifact-backed failure (no junit artifacts)
09:41:08	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 24553e5	unknown	failed	#6670 (open)	1	non-artifact-backed failure (no junit artifacts)
09:39:05	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c9b8452	unknown	failed	#6584 (open)	1	non-artifact-backed failure (no junit artifacts)
09:34:45	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c9b8452	unknown	failed	#6584 (open)	1	non-artifact-backed failure (no junit artifacts)
09:32:33	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 515e34b	unknown	failed	#6583 (open)	1	non-artifact-backed failure (no junit artifacts)
09:29:48	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 7eff23c	alert, e2e	failed	#6575 (open)	2	e2e (1) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 09:29:48 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-6pb5wbpxgt8x/cluster-candidate-4-21-mvzhwr should provision Unexpected error: <fmt.wrapError \| 0xc0009d0280>: failed to create HCP cluster cluster-candidate-4-21-mvzhwr: failed waiting for cluster="cluster-candidate-4-21-mvzhwr" in resourcegroup="rg-candidate-4-21-6pb5wbpxgt8x" to finish creating: GET https://rp.j0831744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/93a57e4a-983c-402e-a03b-d4f6bdacf351 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/93a57e4a-983c-402e-a03b-d4f6bdacf351", "name": "93a57e4a-983c-402e-a03b-d4f6bdacf351", "status": "Failed", "startTime": "2026-08-25T10:41:58.090010693Z", "endTime": "2026-08-25T11:01:01.195580215Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-6pb5wbpxgt8x/cluster-candidate-4-21-mvzhwr should provision Unexpected error: <fmt.wrapError \| 0xc0009d0280>: failed to create HCP cluster cluster-candidate-4-21-mvzhwr: failed waiting for cluster="cluster-candidate-4-21-mvzhwr" in resourcegroup="rg-candidate-4-21-6pb5wbpxgt8x" to finish creating: GET https://rp.j0831744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/93a57e4a-983c-402e-a03b-d4f6bdacf351 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/93a57e4a-983c-402e-a03b-d4f6bdacf351", "name": "93a57e4a-983c-402e-a03b-d4f6bdacf351", "status": "Failed", "startTime": "2026-08-25T10:41:58.090010693Z", "endTime": "2026-08-25T11:01:01.195580215Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [svc] FrontendPathMedianLatency fired 09:29:48 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-25T10:44:13Z Ended: 2026-08-25T11:18:16Z Severity: Sev3 Labels: alertname="FrontendPathMedianLatency", cluster="ci01-j0831744-svc", component="frontend", method="put", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning" Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j0831744-svc).
09:27:11	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c9b8452	unknown	failed	#6584 (open)	1	non-artifact-backed failure (no junit artifacts)
09:26:19	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 476ea91 · merge 476ea91	unknown	failed	#6673 (merged)	1	non-artifact-backed failure (no junit artifacts)
09:26:03	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 24553e5	unknown	failed	#6670 (open)	1	non-artifact-backed failure (no junit artifacts)
09:25:20	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 24553e5	unknown	failed	#6670 (open)	1	non-artifact-backed failure (no junit artifacts)
09:22:01	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 5fdef9d	unknown	failed	#6364 (open)	1	non-artifact-backed failure (no junit artifacts)
09:16:21	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 669f42e	unknown	failed	#6618 (open)	1	non-artifact-backed failure (no junit artifacts)
09:15:29	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 669f42e	unknown	failed	#6618 (open)	1	non-artifact-backed failure (no junit artifacts)
09:11:07	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 24553e5	unknown	failed	#6670 (open)	1	non-artifact-backed failure (no junit artifacts)
09:11:01	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 22dc550	unknown	failed	#6672 (open)	1	non-artifact-backed failure (no junit artifacts)
09:08:37	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 87c9d3c	unknown	failed	#6662 (open)	1	non-artifact-backed failure (no junit artifacts)
09:04:06	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 669f42e	unknown	failed	#6618 (open)	1	non-artifact-backed failure (no junit artifacts)
09:03:22	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 24553e5	unknown	failed	#6670 (open)	1	non-artifact-backed failure (no junit artifacts)
08:57:10	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 80d6445	unknown	failed	#6670 (open)	1	non-artifact-backed failure (no junit artifacts)
08:49:54	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 80d6445	unknown	failed	#6670 (open)	1	non-artifact-backed failure (no junit artifacts)
08:42:05	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 80d6445	unknown	failed	#6670 (open)	1	non-artifact-backed failure (no junit artifacts)
08:38:31	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head aecbaba · merge aecbaba	unknown	failed	#6671 (merged)	1	non-artifact-backed failure (no junit artifacts)
08:35:30	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 80d6445	unknown	failed	#6670 (open)	1	non-artifact-backed failure (no junit artifacts)
08:31:26	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 80d6445	unknown	failed	#6670 (open)	1	non-artifact-backed failure (no junit artifacts)
08:15:32	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 669f42e	unknown	failed	#6618 (open)	1	non-artifact-backed failure (no junit artifacts)
08:14:15	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 515e34b	unknown	failed	#6583 (open)	1	non-artifact-backed failure (no junit artifacts)
08:11:47	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 515e34b	unknown	failed	#6583 (open)	1	non-artifact-backed failure (no junit artifacts)
08:07:18	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 3c81c74	unknown	failed	#6645 (open)	1	non-artifact-backed failure (no junit artifacts)
08:07:15	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 515e34b	unknown	failed	#6583 (open)	1	non-artifact-backed failure (no junit artifacts)
07:54:11	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 515e34b	unknown	failed	#6583 (open)	1	non-artifact-backed failure (no junit artifacts)
07:43:16	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 669f42e	unknown	failed	#6618 (open)	1	non-artifact-backed failure (no junit artifacts)
07:08:13	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 5c0ce63	e2e	failed	#6669 (open)	1	e2e (1) at least one schedule should have a LastBackupTime 07:08:13 UTC · failed at e2e · test SRE can pause schedules to stop backup execution for an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:159]: Timed out after 660.001s. at least one schedule should have a LastBackupTime Expected <bool>: ... to be true fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:159]: Timed out after 660.001s. at least one schedule should have a LastBackupTime Expected <bool>: ... to be true
07:07:02	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c403afe	alert, e2e	failed	#6638 (open)	4	e2e (1) ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 07:07:02 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--kqf9svj6lsq5/cluster-hs-pre-xwpsbd should provision Unexpected error: <fmt.wrapError \| 0xc000e8a8a0>: failed to create HCP cluster cluster-hs-pre-xwpsbd: failed waiting for cluster="cluster-hs-pre-xwpsbd" in resourcegroup="rg-hs-pre--kqf9svj6lsq5" to finish creating: GET https://rp.j6292864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0314e9a-f076-4160-9b33-87341813da0e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0314e9a-f076-4160-9b33-87341813da0e", "name": "f0314e9a-f076-4160-9b33-87341813da0e", "status": "Failed", "startTime": "2026-08-25T07:56:21.373383208Z", "endTime": "2026-08-25T08:15:30.901546963Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--kqf9svj6lsq5/cluster-hs-pre-xwpsbd should provision Unexpected error: <fmt.wrapError \| 0xc000e8a8a0>: failed to create HCP cluster cluster-hs-pre-xwpsbd: failed waiting for cluster="cluster-hs-pre-xwpsbd" in resourcegroup="rg-hs-pre--kqf9svj6lsq5" to finish creating: GET https://rp.j6292864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0314e9a-f076-4160-9b33-87341813da0e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0314e9a-f076-4160-9b33-87341813da0e", "name": "f0314e9a-f076-4160-9b33-87341813da0e", "status": "Failed", "startTime": "2026-08-25T07:56:21.373383208Z", "endTime": "2026-08-25T08:15:30.901546963Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (3) alert [hcp] KubeDeploymentReplicasMismatch fired 07:07:02 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-25T08:31:29Z Ended: 2026-08-25T08:38:27Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6292864-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.76:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6292864/providers/microsoft.monitor/accounts/hcps-j6292864", namespace="ocm-arohcpci01-2sdfp9pn2ipl52p137025ph1c09o5rmr-m7j4d7m1x6s1n9o", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-rzqj4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2sdfp9pn2ipl52p137025ph1c09o5rmr-m7j4d7m1x6s1n9o/capi-provider has not matched the expected number of replicas for longer than 30 minutes. alert [hcp] KubeStatefulSetReplicasMismatch fired 07:07:02 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-08-25T08:17:26Z Ended: 2026-08-25T08:33:27Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j6292864-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.76:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6292864/providers/microsoft.monitor/accounts/hcps-j6292864", namespace="ocm-arohcpci01-2sdfp9pn2ipl52p137025ph1c09o5rmr-m7j4d7m1x6s1n9o", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-rzqj4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2sdfp9pn2ipl52p137025ph1c09o5rmr-m7j4d7m1x6s1n9o/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 2: State: Resolved Started: 2026-08-25T08:17:26Z Ended: 2026-08-25T08:36:25Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j6292864-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.76:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6292864/providers/microsoft.monitor/accounts/hcps-j6292864", namespace="ocm-arohcpci01-2sdfp9pn2ipl52p137025ph1c09o5rmr-m7j4d7m1x6s1n9o", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-rzqj4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2sdfp9pn2ipl52p137025ph1c09o5rmr-m7j4d7m1x6s1n9o/etcd has not matched the expected number of replicas for longer than 15 minutes. alert [hcp] KubePodCrashLooping fired 07:07:02 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-08-25T08:19:22Z Ended: 2026-08-25T08:37:23Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j6292864-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.76:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6292864/providers/microsoft.monitor/accounts/hcps-j6292864", namespace="ocm-arohcpci01-2sdfp9pn2ipl52p137025ph1c09o5rmr-m7j4d7m1x6s1n9o", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="8f2abcc4-a7f6-4094-8ae9-551ab00643de" Description: Pod ocm-arohcpci01-2sdfp9pn2ipl52p137025ph1c09o5rmr-m7j4d7m1x6s1n9o/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Resolved Started: 2026-08-25T08:19:22Z Ended: 2026-08-25T08:36:25Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j6292864-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.76:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6292864/providers/microsoft.monitor/accounts/hcps-j6292864", namespace="ocm-arohcpci01-2sdfp9pn2ipl52p137025ph1c09o5rmr-m7j4d7m1x6s1n9o", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="8f2abcc4-a7f6-4094-8ae9-551ab00643de" Description: Pod ocm-arohcpci01-2sdfp9pn2ipl52p137025ph1c09o5rmr-m7j4d7m1x6s1n9o/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
07:07:01	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c9b8452	n/a	passed	#6584 (open)	0	n/a
07:06:59	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 87c9d3c	n/a	passed	#6662 (open)	0	n/a
07:06:55	pull-ci-Azure-ARO-HCP-main-e2e-parallel head d5a9f48	alert	failed	#6648 (open)	1	alert (1) alert [svc] BackendControllerRetryHotLoop fired 07:06:55 UTC · failed at alert · test [aro-hcp-observability] [svc] alert BackendControllerRetryHotLoop does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-25T08:22:04Z Ended: 2026-08-25T08:51:02Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j8623232-svc", component="backend", name="observemanagedresourcegroup", severity="warning" Description: Backend controller workqueue observemanagedresourcegroup has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work.
07:06:53	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c6d4678	alert, e2e	failed	#6629 (open)	3	e2e (2) ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift 07:06:53 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc0014fa7a0>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-5wf7vp46fpk2" to finish creating: GET https://rp.j6249984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eb3dd7de-2ea8-41b7-8963-6ca228011069 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eb3dd7de-2ea8-41b7-8963-6ca228011069", "name": "eb3dd7de-2ea8-41b7-8963-6ca228011069", "status": "Failed", "startTime": "2026-08-25T08:07:43.594897017Z", "endTime": "2026-08-25T08:26:51.952575117Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc0014fa7a0>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-5wf7vp46fpk2" to finish creating: GET https://rp.j6249984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eb3dd7de-2ea8-41b7-8963-6ca228011069 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eb3dd7de-2ea8-41b7-8963-6ca228011069", "name": "eb3dd7de-2ea8-41b7-8963-6ca228011069", "status": "Failed", "startTime": "2026-08-25T08:07:43.594897017Z", "endTime": "2026-08-25T08:26:51.952575117Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 07:06:53 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc000ec0ac0>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-8pcm786z9kl7" to finish creating: GET https://rp.j6249984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/07285d13-3ea1-4146-88cc-72658c636e9b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/07285d13-3ea1-4146-88cc-72658c636e9b", "name": "07285d13-3ea1-4146-88cc-72658c636e9b", "status": "Failed", "startTime": "2026-08-25T08:06:31.910071805Z", "endTime": "2026-08-25T08:25:31.930786343Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc000ec0ac0>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-8pcm786z9kl7" to finish creating: GET https://rp.j6249984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/07285d13-3ea1-4146-88cc-72658c636e9b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/07285d13-3ea1-4146-88cc-72658c636e9b", "name": "07285d13-3ea1-4146-88cc-72658c636e9b", "status": "Failed", "startTime": "2026-08-25T08:06:31.910071805Z", "endTime": "2026-08-25T08:25:31.930786343Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 07:06:53 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 5 time(s) (1 unknown, 4 known) Firing 1: State: Resolved Started: 2026-08-25T08:17:44Z Ended: 2026-08-25T08:24:51Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j6249984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdfu24s4pvl5a530q5gqgms7ekvmjjt-v20260901", pod="router-f9564f577-ntlvg", severity="warning" Description: Pod ocm-arohcpci01-2sdfu24s4pvl5a530q5gqgms7ekvmjjt-v20260901/router-f9564f577-ntlvg has been in a non-ready state for longer than 5 minutes.
06:51:54	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 7eff23c	e2e	failed	★#6575 (open)	3	e2e (3) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 06:51:54 UTC · failed at e2e · test FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc00110e740>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-ghjz2jzcblzs" to finish creating: GET https://rp.j7480448.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/556c0413-25c1-4420-86cd-b5562eb5b2ba -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/556c0413-25c1-4420-86cd-b5562eb5b2ba", "name": "556c0413-25c1-4420-86cd-b5562eb5b2ba", "status": "Failed", "startTime": "2026-08-25T07:43:34.764859985Z", "endTime": "2026-08-25T08:02:41.010331478Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc00110e740>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-ghjz2jzcblzs" to finish creating: GET https://rp.j7480448.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/556c0413-25c1-4420-86cd-b5562eb5b2ba -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/556c0413-25c1-4420-86cd-b5562eb5b2ba", "name": "556c0413-25c1-4420-86cd-b5562eb5b2ba", "status": "Failed", "startTime": "2026-08-25T07:43:34.764859985Z", "endTime": "2026-08-25T08:02:41.010331478Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift 06:51:54 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc000fa85c0>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-4pdvp7d98cjv" to finish creating: GET https://rp.j7480448.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f38bbf22-3f09-449b-92bd-a2c77dd73dbe -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f38bbf22-3f09-449b-92bd-a2c77dd73dbe", "name": "f38bbf22-3f09-449b-92bd-a2c77dd73dbe", "status": "Failed", "startTime": "2026-08-25T07:44:06.292489025Z", "endTime": "2026-08-25T08:03:10.8621061Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc000fa85c0>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-4pdvp7d98cjv" to finish creating: GET https://rp.j7480448.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f38bbf22-3f09-449b-92bd-a2c77dd73dbe -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f38bbf22-3f09-449b-92bd-a2c77dd73dbe", "name": "f38bbf22-3f09-449b-92bd-a2c77dd73dbe", "status": "Failed", "startTime": "2026-08-25T07:44:06.292489025Z", "endTime": "2026-08-25T08:03:10.8621061Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred query-basic: command failed: exit status 1 06:51:54 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.001s. Expected success, but got an error: <errors.errorString \| 0xc0006c4270>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[08:07:56.002]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T08:07:43.2162872Z);\nlet _endTime = datetime(2026-08-25T08:07:43.2162878Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-nc7qs5pbjx25'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T08:07:55.9842316Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;f956ea66-fdc4-476d-90bd-06769a04a23f\",\r\n \"activityId\": \"93240ce7-344d-4b28-a6d5-bc7c0d41eeab\",\r\n \"subActivityId\": \"daa37bf3-b9a2-4c43-9bab-6f0f919d8ffc\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"2d5a9b37-61e4-4d07-80a4-b16788917aef\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;f956ea66-fdc4-476d-90bd-06769a04a23f ARID=93240ce7-344d-4b28-a6d5-bc7c0d41eeab \u003e GW.Http.CallContext/93240ce7-344d-4b28-a6d5-bc7c0d41eeab \u003e GW.ExecuteQuery/2d5a9b37-61e4-4d07-80a4-b16788917aef \u003e DN.FE.ExecuteQuery/daa37bf3-b9a2-4c43-9bab-6f0f919d8ffc)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[08:07:56.054]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T08:07:43.2162872Z);\nlet _endTime = datetime(2026-08-25T08:07:43.2162878Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-nc7qs5pbjx25'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T08:07:55.9842316Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;f956ea66-fdc4-476d-90bd-06769a04a23f\",\r\n \"activityId\": \"93240ce7-344d-4b28-a6d5-bc7c0d41eeab\",\r\n \"subActivityId\": \"daa37bf3-b9a2-4c43-9bab-6f0f919d8ffc\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"2d5a9b37-61e4-4d07-80a4-b16788917aef\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;f956ea66-fdc4-476d-90bd-06769a04a23f ARID=93240ce7-344d-4b28-a6d5-bc7c0d41eeab \u003e GW.Http.CallContext/93240ce7-344d-4b28-a6d5-bc7c0d41eeab \u003e GW.ExecuteQuery/2d5a9b37-61e4-4d07-80a4-b16788917aef \u003e DN.FE.ExecuteQuery/daa37bf3-b9a2-4c43-9bab-6f0f919d8ffc)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[08:08:04.153]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T08:07:56.1417247Z);\nlet _endTime = datetime(2026-08-25T08:07:56.1417253Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-nc7qs5pbjx25'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T08:08:04.1346416Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;02b7ec99-eb74-4396-80fc-e477f9f18b42\",\r\n \"activityId\": \"5170d421-679a-4a48-b29b-9a996c74dde4\",\r\n \"subActivityId\": \"d37306ab-e852-439a-bb92-4f7c817288f6\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"36329843-81dc-4dc4-aa28-c56ee7342c50\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;02b7ec99-eb74-4396-80fc-e477f9f18b42 ARID=5170d421-679a-4a48-b29b-9a996c74dde4 \u003e GW.Http.CallContext/5170d421-679a-4a48-b29b-9a996c74dde4 \u003e GW.ExecuteQuery/36329843-81dc-4dc4-aa28-c56ee7342c50 \u003e DN.FE.ExecuteQuery/d37306ab-e852-439a-bb92-4f7c817288f6)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[08:08:04.206]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T08:07:56.1417247Z);\nlet _endTime = datetime(2026-08-25T08:07:56.1417253Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-nc7qs5pbjx25'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T08:08:04.1346416Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;02b7ec99-eb74-4396-80fc-e477f9f18b42\",\r\n \"activityId\": \"5170d421-679a-4a48-b29b-9a996c74dde4\",\r\n \"subActivityId\": \"d37306ab-e852-439a-bb92-4f7c817288f6\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"36329843-81dc-4dc4-aa28-c56ee7342c50\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;02b7ec99-eb74-4396-80fc-e477f9f18b42 ARID=5170d421-679a-4a48-b29b-9a996c74dde4 \u003e GW.Http.CallContext/5170d421-679a-4a48-b29b-9a996c74dde4 \u003e GW.ExecuteQuery/36329843-81dc-4dc4-aa28-c56ee7342c50 \u003e DN.FE.ExecuteQuery/d37306ab-e852-439a-bb92-4f7c817288f6)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[08:08:15.998]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T08:08:07.6782331Z);\nlet _endTime = datetime(2026-08-25T08:08:07.6782337Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-nc7qs5pbjx25'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T08:08:15.9798710Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a2ee4734-5cfc-4e05-bbc1-57b733b1306b\",\r\n \"activityId\": \"c0534e85-0700-4932-b07d-9b9e7af63995\",\r\n \"subActivityId\": \"abf8d736-f9c7-4683-a287-d5c9f53d5b71\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"54a6cff9-747f-4ac5-8f21-5804ee2d36e8\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a2ee4734-5cfc-4e05-bbc1-57b733b1306b ARID=c0534e85-0700-4932-b07d-9b9e7af63995 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f52d0643-2180-4672-941a-2aa83b28c631 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f52d0643-2180-4672-941a-2aa83b28c631 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/54a6cff9-747f-4ac5-8f21-5804ee2d36e8 \u003e DN.FE.ExecuteQuery/abf8d736-f9c7-4683-a287-d5c9f53d5b71)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[08:08:16.102]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T08:08:07.6782331Z);\nlet _endTime = datetime(2026-08-25T08:08:07.6782337Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-nc7qs5pbjx25'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T08:08:15.9798710Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a2ee4734-5cfc-4e05-bbc1-57b733b1306b\",\r\n \"activityId\": \"c0534e85-0700-4932-b07d-9b9e7af63995\",\r\n \"subActivityId\": \"abf8d736-f9c7-4683-a287-d5c9f53d5b71\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"54a6cff9-747f-4ac5-8f21-5804ee2d36e8\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a2ee4734-5cfc-4e05-bbc1-57b733b1306b ARID=c0534e85-0700-4932-b07d-9b9e7af63995 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f52d0643-2180-4672-941a-2aa83b28c631 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f52d0643-2180-4672-941a-2aa83b28c631 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/54a6cff9-747f-4ac5-8f21-5804ee2d36e8 \u003e DN.FE.ExecuteQuery/abf8d736-f9c7-4683-a287-d5c9f53d5b71)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.001s. Expected success, but got an error: <errors.errorString \| 0xc0006c4270>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[08:07:56.002]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T08:07:43.2162872Z);\nlet _endTime = datetime(2026-08-25T08:07:43.2162878Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-nc7qs5pbjx25'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T08:07:55.9842316Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;f956ea66-fdc4-476d-90bd-06769a04a23f\",\r\n \"activityId\": \"93240ce7-344d-4b28-a6d5-bc7c0d41eeab\",\r\n \"subActivityId\": \"daa37bf3-b9a2-4c43-9bab-6f0f919d8ffc\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"2d5a9b37-61e4-4d07-80a4-b16788917aef\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;f956ea66-fdc4-476d-90bd-06769a04a23f ARID=93240ce7-344d-4b28-a6d5-bc7c0d41eeab \u003e GW.Http.CallContext/93240ce7-344d-4b28-a6d5-bc7c0d41eeab \u003e GW.ExecuteQuery/2d5a9b37-61e4-4d07-80a4-b16788917aef \u003e DN.FE.ExecuteQuery/daa37bf3-b9a2-4c43-9bab-6f0f919d8ffc)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[08:07:56.054]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T08:07:43.2162872Z);\nlet _endTime = datetime(2026-08-25T08:07:43.2162878Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-nc7qs5pbjx25'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T08:07:55.9842316Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;f956ea66-fdc4-476d-90bd-06769a04a23f\",\r\n \"activityId\": \"93240ce7-344d-4b28-a6d5-bc7c0d41eeab\",\r\n \"subActivityId\": \"daa37bf3-b9a2-4c43-9bab-6f0f919d8ffc\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"2d5a9b37-61e4-4d07-80a4-b16788917aef\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;f956ea66-fdc4-476d-90bd-06769a04a23f ARID=93240ce7-344d-4b28-a6d5-bc7c0d41eeab \u003e GW.Http.CallContext/93240ce7-344d-4b28-a6d5-bc7c0d41eeab \u003e GW.ExecuteQuery/2d5a9b37-61e4-4d07-80a4-b16788917aef \u003e DN.FE.ExecuteQuery/daa37bf3-b9a2-4c43-9bab-6f0f919d8ffc)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[08:08:04.153]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T08:07:56.1417247Z);\nlet _endTime = datetime(2026-08-25T08:07:56.1417253Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-nc7qs5pbjx25'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T08:08:04.1346416Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;02b7ec99-eb74-4396-80fc-e477f9f18b42\",\r\n \"activityId\": \"5170d421-679a-4a48-b29b-9a996c74dde4\",\r\n \"subActivityId\": \"d37306ab-e852-439a-bb92-4f7c817288f6\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"36329843-81dc-4dc4-aa28-c56ee7342c50\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;02b7ec99-eb74-4396-80fc-e477f9f18b42 ARID=5170d421-679a-4a48-b29b-9a996c74dde4 \u003e GW.Http.CallContext/5170d421-679a-4a48-b29b-9a996c74dde4 \u003e GW.ExecuteQuery/36329843-81dc-4dc4-aa28-c56ee7342c50 \u003e DN.FE.ExecuteQuery/d37306ab-e852-439a-bb92-4f7c817288f6)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[08:08:04.206]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T08:07:56.1417247Z);\nlet _endTime = datetime(2026-08-25T08:07:56.1417253Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-nc7qs5pbjx25'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T08:08:04.1346416Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;02b7ec99-eb74-4396-80fc-e477f9f18b42\",\r\n \"activityId\": \"5170d421-679a-4a48-b29b-9a996c74dde4\",\r\n \"subActivityId\": \"d37306ab-e852-439a-bb92-4f7c817288f6\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"36329843-81dc-4dc4-aa28-c56ee7342c50\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;02b7ec99-eb74-4396-80fc-e477f9f18b42 ARID=5170d421-679a-4a48-b29b-9a996c74dde4 \u003e GW.Http.CallContext/5170d421-679a-4a48-b29b-9a996c74dde4 \u003e GW.ExecuteQuery/36329843-81dc-4dc4-aa28-c56ee7342c50 \u003e DN.FE.ExecuteQuery/d37306ab-e852-439a-bb92-4f7c817288f6)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[08:08:15.998]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T08:08:07.6782331Z);\nlet _endTime = datetime(2026-08-25T08:08:07.6782337Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-nc7qs5pbjx25'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T08:08:15.9798710Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a2ee4734-5cfc-4e05-bbc1-57b733b1306b\",\r\n \"activityId\": \"c0534e85-0700-4932-b07d-9b9e7af63995\",\r\n \"subActivityId\": \"abf8d736-f9c7-4683-a287-d5c9f53d5b71\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"54a6cff9-747f-4ac5-8f21-5804ee2d36e8\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a2ee4734-5cfc-4e05-bbc1-57b733b1306b ARID=c0534e85-0700-4932-b07d-9b9e7af63995 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f52d0643-2180-4672-941a-2aa83b28c631 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f52d0643-2180-4672-941a-2aa83b28c631 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/54a6cff9-747f-4ac5-8f21-5804ee2d36e8 \u003e DN.FE.ExecuteQuery/abf8d736-f9c7-4683-a287-d5c9f53d5b71)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[08:08:16.102]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T08:08:07.6782331Z);\nlet _endTime = datetime(2026-08-25T08:08:07.6782337Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-nc7qs5pbjx25'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T08:08:15.9798710Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a2ee4734-5cfc-4e05-bbc1-57b733b1306b\",\r\n \"activityId\": \"c0534e85-0700-4932-b07d-9b9e7af63995\",\r\n \"subActivityId\": \"abf8d736-f9c7-4683-a287-d5c9f53d5b71\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"54a6cff9-747f-4ac5-8f21-5804ee2d36e8\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a2ee4734-5cfc-4e05-bbc1-57b733b1306b ARID=c0534e85-0700-4932-b07d-9b9e7af63995 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f52d0643-2180-4672-941a-2aa83b28c631 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f52d0643-2180-4672-941a-2aa83b28c631 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/54a6cff9-747f-4ac5-8f21-5804ee2d36e8 \u003e DN.FE.ExecuteQuery/abf8d736-f9c7-4683-a287-d5c9f53d5b71)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ...
06:27:13	pull-ci-Azure-ARO-HCP-main-e2e-parallel head bf87cfe	alert, e2e	failed	★#6658 (open)	7	e2e (6) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 06:27:13 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-5h9dtgrjffg9/cluster-candidate-5-0-6hrjbx should provision Unexpected error: <fmt.wrapError \| 0xc001834220>: failed to create HCP cluster cluster-candidate-5-0-6hrjbx: failed waiting for cluster="cluster-candidate-5-0-6hrjbx" in resourcegroup="rg-candidate-5-0-5h9dtgrjffg9" to finish creating: GET https://rp.j1244288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/47b8e57b-cd76-4c09-a1f3-3c1abc70b222 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/47b8e57b-cd76-4c09-a1f3-3c1abc70b222", "name": "47b8e57b-cd76-4c09-a1f3-3c1abc70b222", "status": "Failed", "startTime": "2026-08-25T07:14:43.287892255Z", "endTime": "2026-08-25T07:33:50.437023697Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-5h9dtgrjffg9/cluster-candidate-5-0-6hrjbx should provision Unexpected error: <fmt.wrapError \| 0xc001834220>: failed to create HCP cluster cluster-candidate-5-0-6hrjbx: failed waiting for cluster="cluster-candidate-5-0-6hrjbx" in resourcegroup="rg-candidate-5-0-5h9dtgrjffg9" to finish creating: GET https://rp.j1244288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/47b8e57b-cd76-4c09-a1f3-3c1abc70b222 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/47b8e57b-cd76-4c09-a1f3-3c1abc70b222", "name": "47b8e57b-cd76-4c09-a1f3-3c1abc70b222", "status": "Failed", "startTime": "2026-08-25T07:14:43.287892255Z", "endTime": "2026-08-25T07:33:50.437023697Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 06:27:13 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc00086b1c0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-xsjvnxt7d74k" to finish creating: GET https://rp.j1244288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e6029c8d-c9f8-41ef-a520-4b8820105630 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e6029c8d-c9f8-41ef-a520-4b8820105630", "name": "e6029c8d-c9f8-41ef-a520-4b8820105630", "status": "Failed", "startTime": "2026-08-25T07:16:14.334902992Z", "endTime": "2026-08-25T07:35:20.361847152Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc00086b1c0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-xsjvnxt7d74k" to finish creating: GET https://rp.j1244288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e6029c8d-c9f8-41ef-a520-4b8820105630 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e6029c8d-c9f8-41ef-a520-4b8820105630", "name": "e6029c8d-c9f8-41ef-a520-4b8820105630", "status": "Failed", "startTime": "2026-08-25T07:16:14.334902992Z", "endTime": "2026-08-25T07:35:20.361847152Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred query-basic: command failed: exit status 1 06:27:13 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.004s. Expected success, but got an error: <errors.errorString \| 0xc000812850>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[07:36:48.911]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T07:36:45.8465920Z);\nlet _endTime = datetime(2026-08-25T07:36:45.8465927Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-vsdwgwkgtdjp'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T07:36:48.8924657Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;bbdb89c0-4fec-404c-bdd6-6d818f1cded1\",\r\n \"activityId\": \"2bf9b4f0-5c94-4ac1-bcf2-b176b70ad672\",\r\n \"subActivityId\": \"403b7745-bbff-4c9f-929d-476fbd66118e\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"e88d573b-aeb8-4a17-9fca-c87f04c6cdd9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;bbdb89c0-4fec-404c-bdd6-6d818f1cded1 ARID=2bf9b4f0-5c94-4ac1-bcf2-b176b70ad672 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/867d2a47-dbd4-435d-8c9b-766c6c523e4e \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/867d2a47-dbd4-435d-8c9b-766c6c523e4e \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/e88d573b-aeb8-4a17-9fca-c87f04c6cdd9 \u003e DN.FE.ExecuteQuery/403b7745-bbff-4c9f-929d-476fbd66118e)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[07:36:48.962]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T07:36:45.8465920Z);\nlet _endTime = datetime(2026-08-25T07:36:45.8465927Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-vsdwgwkgtdjp'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T07:36:48.8924657Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;bbdb89c0-4fec-404c-bdd6-6d818f1cded1\",\r\n \"activityId\": \"2bf9b4f0-5c94-4ac1-bcf2-b176b70ad672\",\r\n \"subActivityId\": \"403b7745-bbff-4c9f-929d-476fbd66118e\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"e88d573b-aeb8-4a17-9fca-c87f04c6cdd9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;bbdb89c0-4fec-404c-bdd6-6d818f1cded1 ARID=2bf9b4f0-5c94-4ac1-bcf2-b176b70ad672 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/867d2a47-dbd4-435d-8c9b-766c6c523e4e \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/867d2a47-dbd4-435d-8c9b-766c6c523e4e \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/e88d573b-aeb8-4a17-9fca-c87f04c6cdd9 \u003e DN.FE.ExecuteQuery/403b7745-bbff-4c9f-929d-476fbd66118e)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[07:36:51.855]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T07:36:49.0279421Z);\nlet _endTime = datetime(2026-08-25T07:36:49.0279424Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-vsdwgwkgtdjp'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T07:36:51.8360877Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;d44a550b-983d-459e-b914-0653df2e0e3b\",\r\n \"activityId\": \"f4d03f54-2298-4d29-b8e3-9d984c0bd2b9\",\r\n \"subActivityId\": \"60e487db-9b15-48f2-a301-b66685f17ff4\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"4dc8f887-d2c7-4ef0-ad29-017a6723ebe4\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;d44a550b-983d-459e-b914-0653df2e0e3b ARID=f4d03f54-2298-4d29-b8e3-9d984c0bd2b9 \u003e GW.Http.CallContext/f4d03f54-2298-4d29-b8e3-9d984c0bd2b9 \u003e GW.ExecuteQuery/4dc8f887-d2c7-4ef0-ad29-017a6723ebe4 \u003e DN.FE.ExecuteQuery/60e487db-9b15-48f2-a301-b66685f17ff4)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[07:36:51.908]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T07:36:49.0279421Z);\nlet _endTime = datetime(2026-08-25T07:36:49.0279424Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-vsdwgwkgtdjp'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T07:36:51.8360877Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;d44a550b-983d-459e-b914-0653df2e0e3b\",\r\n \"activityId\": \"f4d03f54-2298-4d29-b8e3-9d984c0bd2b9\",\r\n \"subActivityId\": \"60e487db-9b15-48f2-a301-b66685f17ff4\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"4dc8f887-d2c7-4ef0-ad29-017a6723ebe4\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;d44a550b-983d-459e-b914-0653df2e0e3b ARID=f4d03f54-2298-4d29-b8e3-9d984c0bd2b9 \u003e GW.Http.CallContext/f4d03f54-2298-4d29-b8e3-9d984c0bd2b9 \u003e GW.ExecuteQuery/4dc8f887-d2c7-4ef0-ad29-017a6723ebe4 \u003e DN.FE.ExecuteQuery/60e487db-9b15-48f2-a301-b66685f17ff4)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[07:36:58.311]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T07:36:55.2926733Z);\nlet _endTime = datetime(2026-08-25T07:36:55.2926740Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-vsdwgwkgtdjp'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T07:36:58.2926720Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;da67d458-be70-4b7e-b6dd-55d1adf76958\",\r\n \"activityId\": \"5168b874-e371-442d-96d4-634b404e01b7\",\r\n \"subActivityId\": \"a4dec94c-d4be-425e-8443-3e3dba0c4a46\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"33d6b2a1-bd08-472a-87a5-98f6eafd09fb\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;da67d458-be70-4b7e-b6dd-55d1adf76958 ARID=5168b874-e371-442d-96d4-634b404e01b7 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/d6496d65-ab35-4ce7-ab25-0229e9045b36 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/d6496d65-ab35-4ce7-ab25-0229e9045b36 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/33d6b2a1-bd08-472a-87a5-98f6eafd09fb \u003e DN.FE.ExecuteQuery/a4dec94c-d4be-425e-8443-3e3dba0c4a46)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[07:36:58.405]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T07:36:55.2926733Z);\nlet _endTime = datetime(2026-08-25T07:36:55.2926740Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-vsdwgwkgtdjp'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T07:36:58.2926720Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;da67d458-be70-4b7e-b6dd-55d1adf76958\",\r\n \"activityId\": \"5168b874-e371-442d-96d4-634b404e01b7\",\r\n \"subActivityId\": \"a4dec94c-d4be-425e-8443-3e3dba0c4a46\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"33d6b2a1-bd08-472a-87a5-98f6eafd09fb\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;da67d458-be70-4b7e-b6dd-55d1adf76958 ARID=5168b874-e371-442d-96d4-634b404e01b7 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/d6496d65-ab35-4ce7-ab25-0229e9045b36 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/d6496d65-ab35-4ce7-ab25-0229e9045b36 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/33d6b2a1-bd08-472a-87a5-98f6eafd09fb \u003e DN.FE.ExecuteQuery/a4dec94c-d4be-425e-8443-3e3dba0c4a46)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.004s. Expected success, but got an error: <errors.errorString \| 0xc000812850>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[07:36:48.911]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T07:36:45.8465920Z);\nlet _endTime = datetime(2026-08-25T07:36:45.8465927Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-vsdwgwkgtdjp'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T07:36:48.8924657Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;bbdb89c0-4fec-404c-bdd6-6d818f1cded1\",\r\n \"activityId\": \"2bf9b4f0-5c94-4ac1-bcf2-b176b70ad672\",\r\n \"subActivityId\": \"403b7745-bbff-4c9f-929d-476fbd66118e\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"e88d573b-aeb8-4a17-9fca-c87f04c6cdd9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;bbdb89c0-4fec-404c-bdd6-6d818f1cded1 ARID=2bf9b4f0-5c94-4ac1-bcf2-b176b70ad672 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/867d2a47-dbd4-435d-8c9b-766c6c523e4e \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/867d2a47-dbd4-435d-8c9b-766c6c523e4e \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/e88d573b-aeb8-4a17-9fca-c87f04c6cdd9 \u003e DN.FE.ExecuteQuery/403b7745-bbff-4c9f-929d-476fbd66118e)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[07:36:48.962]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T07:36:45.8465920Z);\nlet _endTime = datetime(2026-08-25T07:36:45.8465927Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-vsdwgwkgtdjp'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T07:36:48.8924657Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;bbdb89c0-4fec-404c-bdd6-6d818f1cded1\",\r\n \"activityId\": \"2bf9b4f0-5c94-4ac1-bcf2-b176b70ad672\",\r\n \"subActivityId\": \"403b7745-bbff-4c9f-929d-476fbd66118e\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"e88d573b-aeb8-4a17-9fca-c87f04c6cdd9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;bbdb89c0-4fec-404c-bdd6-6d818f1cded1 ARID=2bf9b4f0-5c94-4ac1-bcf2-b176b70ad672 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/867d2a47-dbd4-435d-8c9b-766c6c523e4e \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/867d2a47-dbd4-435d-8c9b-766c6c523e4e \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/e88d573b-aeb8-4a17-9fca-c87f04c6cdd9 \u003e DN.FE.ExecuteQuery/403b7745-bbff-4c9f-929d-476fbd66118e)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[07:36:51.855]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T07:36:49.0279421Z);\nlet _endTime = datetime(2026-08-25T07:36:49.0279424Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-vsdwgwkgtdjp'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T07:36:51.8360877Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;d44a550b-983d-459e-b914-0653df2e0e3b\",\r\n \"activityId\": \"f4d03f54-2298-4d29-b8e3-9d984c0bd2b9\",\r\n \"subActivityId\": \"60e487db-9b15-48f2-a301-b66685f17ff4\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"4dc8f887-d2c7-4ef0-ad29-017a6723ebe4\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;d44a550b-983d-459e-b914-0653df2e0e3b ARID=f4d03f54-2298-4d29-b8e3-9d984c0bd2b9 \u003e GW.Http.CallContext/f4d03f54-2298-4d29-b8e3-9d984c0bd2b9 \u003e GW.ExecuteQuery/4dc8f887-d2c7-4ef0-ad29-017a6723ebe4 \u003e DN.FE.ExecuteQuery/60e487db-9b15-48f2-a301-b66685f17ff4)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[07:36:51.908]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T07:36:49.0279421Z);\nlet _endTime = datetime(2026-08-25T07:36:49.0279424Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-vsdwgwkgtdjp'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T07:36:51.8360877Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;d44a550b-983d-459e-b914-0653df2e0e3b\",\r\n \"activityId\": \"f4d03f54-2298-4d29-b8e3-9d984c0bd2b9\",\r\n \"subActivityId\": \"60e487db-9b15-48f2-a301-b66685f17ff4\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"4dc8f887-d2c7-4ef0-ad29-017a6723ebe4\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;d44a550b-983d-459e-b914-0653df2e0e3b ARID=f4d03f54-2298-4d29-b8e3-9d984c0bd2b9 \u003e GW.Http.CallContext/f4d03f54-2298-4d29-b8e3-9d984c0bd2b9 \u003e GW.ExecuteQuery/4dc8f887-d2c7-4ef0-ad29-017a6723ebe4 \u003e DN.FE.ExecuteQuery/60e487db-9b15-48f2-a301-b66685f17ff4)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[07:36:58.311]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T07:36:55.2926733Z);\nlet _endTime = datetime(2026-08-25T07:36:55.2926740Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-vsdwgwkgtdjp'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T07:36:58.2926720Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;da67d458-be70-4b7e-b6dd-55d1adf76958\",\r\n \"activityId\": \"5168b874-e371-442d-96d4-634b404e01b7\",\r\n \"subActivityId\": \"a4dec94c-d4be-425e-8443-3e3dba0c4a46\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"33d6b2a1-bd08-472a-87a5-98f6eafd09fb\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;da67d458-be70-4b7e-b6dd-55d1adf76958 ARID=5168b874-e371-442d-96d4-634b404e01b7 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/d6496d65-ab35-4ce7-ab25-0229e9045b36 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/d6496d65-ab35-4ce7-ab25-0229e9045b36 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/33d6b2a1-bd08-472a-87a5-98f6eafd09fb \u003e DN.FE.ExecuteQuery/a4dec94c-d4be-425e-8443-3e3dba0c4a46)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[07:36:58.405]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T07:36:55.2926733Z);\nlet _endTime = datetime(2026-08-25T07:36:55.2926740Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-vsdwgwkgtdjp'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T07:36:58.2926720Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;da67d458-be70-4b7e-b6dd-55d1adf76958\",\r\n \"activityId\": \"5168b874-e371-442d-96d4-634b404e01b7\",\r\n \"subActivityId\": \"a4dec94c-d4be-425e-8443-3e3dba0c4a46\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"33d6b2a1-bd08-472a-87a5-98f6eafd09fb\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;da67d458-be70-4b7e-b6dd-55d1adf76958 ARID=5168b874-e371-442d-96d4-634b404e01b7 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/d6496d65-ab35-4ce7-ab25-0229e9045b36 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/d6496d65-ab35-4ce7-ab25-0229e9045b36 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/33d6b2a1-bd08-472a-87a5-98f6eafd09fb \u003e DN.FE.ExecuteQuery/a4dec94c-d4be-425e-8443-3e3dba0c4a46)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 06:27:13 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc000e409c0>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-hnsmj56zlr79" to finish creating: GET https://rp.j1244288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c2d4a286-1d08-44dc-8c95-6b2184c8c829 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c2d4a286-1d08-44dc-8c95-6b2184c8c829", "name": "c2d4a286-1d08-44dc-8c95-6b2184c8c829", "status": "Failed", "startTime": "2026-08-25T07:16:14.438595425Z", "endTime": "2026-08-25T07:35:20.381767883Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc000e409c0>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-hnsmj56zlr79" to finish creating: GET https://rp.j1244288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c2d4a286-1d08-44dc-8c95-6b2184c8c829 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c2d4a286-1d08-44dc-8c95-6b2184c8c829", "name": "c2d4a286-1d08-44dc-8c95-6b2184c8c829", "status": "Failed", "startTime": "2026-08-25T07:16:14.438595425Z", "endTime": "2026-08-25T07:35:20.381767883Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 06:27:13 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc0001ab3e0>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-rcn4gmx4mkc4" to finish creating: GET https://rp.j1244288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2a53eabb-5825-4532-a8be-6f9a8d4224ca -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2a53eabb-5825-4532-a8be-6f9a8d4224ca", "name": "2a53eabb-5825-4532-a8be-6f9a8d4224ca", "status": "Failed", "startTime": "2026-08-25T07:14:44.590970562Z", "endTime": "2026-08-25T07:33:50.362043926Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc0001ab3e0>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-rcn4gmx4mkc4" to finish creating: GET https://rp.j1244288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2a53eabb-5825-4532-a8be-6f9a8d4224ca -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2a53eabb-5825-4532-a8be-6f9a8d4224ca", "name": "2a53eabb-5825-4532-a8be-6f9a8d4224ca", "status": "Failed", "startTime": "2026-08-25T07:14:44.590970562Z", "endTime": "2026-08-25T07:33:50.362043926Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 06:27:13 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-9sc24ws2kvm7/cluster-candidate-4-22-tjqkdx should provision Unexpected error: <fmt.wrapError \| 0xc000ee8560>: failed to create HCP cluster cluster-candidate-4-22-tjqkdx: failed waiting for cluster="cluster-candidate-4-22-tjqkdx" in resourcegroup="rg-candidate-4-22-9sc24ws2kvm7" to finish creating: GET https://rp.j1244288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f7e0ad4e-0d4b-40fd-8921-744d4dcbc017 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f7e0ad4e-0d4b-40fd-8921-744d4dcbc017", "name": "f7e0ad4e-0d4b-40fd-8921-744d4dcbc017", "status": "Failed", "startTime": "2026-08-25T07:14:43.689975191Z", "endTime": "2026-08-25T07:33:50.401875542Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-9sc24ws2kvm7/cluster-candidate-4-22-tjqkdx should provision Unexpected error: <fmt.wrapError \| 0xc000ee8560>: failed to create HCP cluster cluster-candidate-4-22-tjqkdx: failed waiting for cluster="cluster-candidate-4-22-tjqkdx" in resourcegroup="rg-candidate-4-22-9sc24ws2kvm7" to finish creating: GET https://rp.j1244288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f7e0ad4e-0d4b-40fd-8921-744d4dcbc017 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f7e0ad4e-0d4b-40fd-8921-744d4dcbc017", "name": "f7e0ad4e-0d4b-40fd-8921-744d4dcbc017", "status": "Failed", "startTime": "2026-08-25T07:14:43.689975191Z", "endTime": "2026-08-25T07:33:50.401875542Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 06:27:13 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-25T07:28:31Z Ended: 2026-08-25T07:46:35Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1244288-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdf5p07vju636qclvute8eqflcbovqk-n4c7o9t2g4r4v7k", pod="router-bffff87c6-wgpsj", severity="warning" Description: Pod ocm-arohcpci01-2sdf5p07vju636qclvute8eqflcbovqk-n4c7o9t2g4r4v7k/router-bffff87c6-wgpsj has been in a non-ready state for longer than 5 minutes.
06:06:28	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 515e34b	e2e	failed	★#6583 (open)	1	e2e (1) query-basic: command failed: exit status 1 06:06:28 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.003s. Expected success, but got an error: <errors.errorString \| 0xc000a22250>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[06:58:42.889]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T06:58:39.6001491Z);\nlet _endTime = datetime(2026-08-25T06:58:39.6001496Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bm9448kb54lr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T06:58:42.8701809Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;54e3445c-767b-4a53-ab5e-6a8cb12abe5c\",\r\n \"activityId\": \"50929d0b-6732-4e31-a429-a529a01fac9c\",\r\n \"subActivityId\": \"29a9a3a1-3134-4073-9f9f-167f7be9ed6d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"9e96b72c-b3c0-4907-ba16-c5f840899e89\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;54e3445c-767b-4a53-ab5e-6a8cb12abe5c ARID=50929d0b-6732-4e31-a429-a529a01fac9c \u003e GW.Http.CallContext/50929d0b-6732-4e31-a429-a529a01fac9c \u003e GW.ExecuteQuery/9e96b72c-b3c0-4907-ba16-c5f840899e89 \u003e DN.FE.ExecuteQuery/29a9a3a1-3134-4073-9f9f-167f7be9ed6d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[06:58:42.938]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T06:58:39.6001491Z);\nlet _endTime = datetime(2026-08-25T06:58:39.6001496Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bm9448kb54lr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T06:58:42.8701809Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;54e3445c-767b-4a53-ab5e-6a8cb12abe5c\",\r\n \"activityId\": \"50929d0b-6732-4e31-a429-a529a01fac9c\",\r\n \"subActivityId\": \"29a9a3a1-3134-4073-9f9f-167f7be9ed6d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"9e96b72c-b3c0-4907-ba16-c5f840899e89\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;54e3445c-767b-4a53-ab5e-6a8cb12abe5c ARID=50929d0b-6732-4e31-a429-a529a01fac9c \u003e GW.Http.CallContext/50929d0b-6732-4e31-a429-a529a01fac9c \u003e GW.ExecuteQuery/9e96b72c-b3c0-4907-ba16-c5f840899e89 \u003e DN.FE.ExecuteQuery/29a9a3a1-3134-4073-9f9f-167f7be9ed6d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[06:58:46.438]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T06:58:43.0033907Z);\nlet _endTime = datetime(2026-08-25T06:58:43.0033913Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bm9448kb54lr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T06:58:46.4202164Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;2b381053-8eff-40bc-996d-50543376f61a\",\r\n \"activityId\": \"4fca92f9-d3e2-4fa6-81c9-a9fc8513a713\",\r\n \"subActivityId\": \"3821d24b-5b2c-40a4-bcae-6560c92f6b97\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"48e1149d-7cf7-4df9-9a59-15c62ac82ce9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;2b381053-8eff-40bc-996d-50543376f61a ARID=4fca92f9-d3e2-4fa6-81c9-a9fc8513a713 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/431aa304-259b-4063-8333-d98d7ccf1795 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/431aa304-259b-4063-8333-d98d7ccf1795 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/48e1149d-7cf7-4df9-9a59-15c62ac82ce9 \u003e DN.FE.ExecuteQuery/3821d24b-5b2c-40a4-bcae-6560c92f6b97)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[06:58:46.487]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T06:58:43.0033907Z);\nlet _endTime = datetime(2026-08-25T06:58:43.0033913Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bm9448kb54lr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T06:58:46.4202164Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;2b381053-8eff-40bc-996d-50543376f61a\",\r\n \"activityId\": \"4fca92f9-d3e2-4fa6-81c9-a9fc8513a713\",\r\n \"subActivityId\": \"3821d24b-5b2c-40a4-bcae-6560c92f6b97\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"48e1149d-7cf7-4df9-9a59-15c62ac82ce9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;2b381053-8eff-40bc-996d-50543376f61a ARID=4fca92f9-d3e2-4fa6-81c9-a9fc8513a713 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/431aa304-259b-4063-8333-d98d7ccf1795 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/431aa304-259b-4063-8333-d98d7ccf1795 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/48e1149d-7cf7-4df9-9a59-15c62ac82ce9 \u003e DN.FE.ExecuteQuery/3821d24b-5b2c-40a4-bcae-6560c92f6b97)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[06:58:53.091]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T06:58:49.9195659Z);\nlet _endTime = datetime(2026-08-25T06:58:49.9195666Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bm9448kb54lr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T06:58:53.0733729Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;d5d7ce70-22dd-4c8f-9299-1dc49ba137bf\",\r\n \"activityId\": \"8f036878-c750-44d8-94c2-d0ffa48c5023\",\r\n \"subActivityId\": \"f3a261d8-5aa0-45c4-8e2e-c9b4dc64eef5\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"ecd293b9-988c-49f9-988b-2c0d898fd0bb\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;d5d7ce70-22dd-4c8f-9299-1dc49ba137bf ARID=8f036878-c750-44d8-94c2-d0ffa48c5023 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/e8598cfd-9337-4d84-8e85-f63664cea4a8 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/e8598cfd-9337-4d84-8e85-f63664cea4a8 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/ecd293b9-988c-49f9-988b-2c0d898fd0bb \u003e DN.FE.ExecuteQuery/f3a261d8-5aa0-45c4-8e2e-c9b4dc64eef5)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[06:58:53.185]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T06:58:49.9195659Z);\nlet _endTime = datetime(2026-08-25T06:58:49.9195666Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bm9448kb54lr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T06:58:53.0733729Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;d5d7ce70-22dd-4c8f-9299-1dc49ba137bf\",\r\n \"activityId\": \"8f036878-c750-44d8-94c2-d0ffa48c5023\",\r\n \"subActivityId\": \"f3a261d8-5aa0-45c4-8e2e-c9b4dc64eef5\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"ecd293b9-988c-49f9-988b-2c0d898fd0bb\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;d5d7ce70-22dd-4c8f-9299-1dc49ba137bf ARID=8f036878-c750-44d8-94c2-d0ffa48c5023 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/e8598cfd-9337-4d84-8e85-f63664cea4a8 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/e8598cfd-9337-4d84-8e85-f63664cea4a8 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/ecd293b9-988c-49f9-988b-2c0d898fd0bb \u003e DN.FE.ExecuteQuery/f3a261d8-5aa0-45c4-8e2e-c9b4dc64eef5)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.003s. Expected success, but got an error: <errors.errorString \| 0xc000a22250>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[06:58:42.889]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T06:58:39.6001491Z);\nlet _endTime = datetime(2026-08-25T06:58:39.6001496Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bm9448kb54lr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T06:58:42.8701809Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;54e3445c-767b-4a53-ab5e-6a8cb12abe5c\",\r\n \"activityId\": \"50929d0b-6732-4e31-a429-a529a01fac9c\",\r\n \"subActivityId\": \"29a9a3a1-3134-4073-9f9f-167f7be9ed6d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"9e96b72c-b3c0-4907-ba16-c5f840899e89\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;54e3445c-767b-4a53-ab5e-6a8cb12abe5c ARID=50929d0b-6732-4e31-a429-a529a01fac9c \u003e GW.Http.CallContext/50929d0b-6732-4e31-a429-a529a01fac9c \u003e GW.ExecuteQuery/9e96b72c-b3c0-4907-ba16-c5f840899e89 \u003e DN.FE.ExecuteQuery/29a9a3a1-3134-4073-9f9f-167f7be9ed6d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[06:58:42.938]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T06:58:39.6001491Z);\nlet _endTime = datetime(2026-08-25T06:58:39.6001496Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bm9448kb54lr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T06:58:42.8701809Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;54e3445c-767b-4a53-ab5e-6a8cb12abe5c\",\r\n \"activityId\": \"50929d0b-6732-4e31-a429-a529a01fac9c\",\r\n \"subActivityId\": \"29a9a3a1-3134-4073-9f9f-167f7be9ed6d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"9e96b72c-b3c0-4907-ba16-c5f840899e89\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;54e3445c-767b-4a53-ab5e-6a8cb12abe5c ARID=50929d0b-6732-4e31-a429-a529a01fac9c \u003e GW.Http.CallContext/50929d0b-6732-4e31-a429-a529a01fac9c \u003e GW.ExecuteQuery/9e96b72c-b3c0-4907-ba16-c5f840899e89 \u003e DN.FE.ExecuteQuery/29a9a3a1-3134-4073-9f9f-167f7be9ed6d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[06:58:46.438]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T06:58:43.0033907Z);\nlet _endTime = datetime(2026-08-25T06:58:43.0033913Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bm9448kb54lr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T06:58:46.4202164Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;2b381053-8eff-40bc-996d-50543376f61a\",\r\n \"activityId\": \"4fca92f9-d3e2-4fa6-81c9-a9fc8513a713\",\r\n \"subActivityId\": \"3821d24b-5b2c-40a4-bcae-6560c92f6b97\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"48e1149d-7cf7-4df9-9a59-15c62ac82ce9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;2b381053-8eff-40bc-996d-50543376f61a ARID=4fca92f9-d3e2-4fa6-81c9-a9fc8513a713 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/431aa304-259b-4063-8333-d98d7ccf1795 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/431aa304-259b-4063-8333-d98d7ccf1795 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/48e1149d-7cf7-4df9-9a59-15c62ac82ce9 \u003e DN.FE.ExecuteQuery/3821d24b-5b2c-40a4-bcae-6560c92f6b97)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[06:58:46.487]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T06:58:43.0033907Z);\nlet _endTime = datetime(2026-08-25T06:58:43.0033913Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bm9448kb54lr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T06:58:46.4202164Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;2b381053-8eff-40bc-996d-50543376f61a\",\r\n \"activityId\": \"4fca92f9-d3e2-4fa6-81c9-a9fc8513a713\",\r\n \"subActivityId\": \"3821d24b-5b2c-40a4-bcae-6560c92f6b97\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"48e1149d-7cf7-4df9-9a59-15c62ac82ce9\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;2b381053-8eff-40bc-996d-50543376f61a ARID=4fca92f9-d3e2-4fa6-81c9-a9fc8513a713 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/431aa304-259b-4063-8333-d98d7ccf1795 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/431aa304-259b-4063-8333-d98d7ccf1795 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/48e1149d-7cf7-4df9-9a59-15c62ac82ce9 \u003e DN.FE.ExecuteQuery/3821d24b-5b2c-40a4-bcae-6560c92f6b97)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[06:58:53.091]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T06:58:49.9195659Z);\nlet _endTime = datetime(2026-08-25T06:58:49.9195666Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bm9448kb54lr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T06:58:53.0733729Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;d5d7ce70-22dd-4c8f-9299-1dc49ba137bf\",\r\n \"activityId\": \"8f036878-c750-44d8-94c2-d0ffa48c5023\",\r\n \"subActivityId\": \"f3a261d8-5aa0-45c4-8e2e-c9b4dc64eef5\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"ecd293b9-988c-49f9-988b-2c0d898fd0bb\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;d5d7ce70-22dd-4c8f-9299-1dc49ba137bf ARID=8f036878-c750-44d8-94c2-d0ffa48c5023 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/e8598cfd-9337-4d84-8e85-f63664cea4a8 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/e8598cfd-9337-4d84-8e85-f63664cea4a8 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/ecd293b9-988c-49f9-988b-2c0d898fd0bb \u003e DN.FE.ExecuteQuery/f3a261d8-5aa0-45c4-8e2e-c9b4dc64eef5)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[06:58:53.185]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T06:58:49.9195659Z);\nlet _endTime = datetime(2026-08-25T06:58:49.9195666Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bm9448kb54lr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T06:58:53.0733729Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;d5d7ce70-22dd-4c8f-9299-1dc49ba137bf\",\r\n \"activityId\": \"8f036878-c750-44d8-94c2-d0ffa48c5023\",\r\n \"subActivityId\": \"f3a261d8-5aa0-45c4-8e2e-c9b4dc64eef5\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"ecd293b9-988c-49f9-988b-2c0d898fd0bb\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;d5d7ce70-22dd-4c8f-9299-1dc49ba137bf ARID=8f036878-c750-44d8-94c2-d0ffa48c5023 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/e8598cfd-9337-4d84-8e85-f63664cea4a8 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/e8598cfd-9337-4d84-8e85-f63664cea4a8 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/ecd293b9-988c-49f9-988b-2c0d898fd0bb \u003e DN.FE.ExecuteQuery/f3a261d8-5aa0-45c4-8e2e-c9b4dc64eef5)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ...
06:04:04	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 9ec6505	alert, e2e	failed	★#6433 (open)	5	e2e (4) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 06:04:04 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc0012c2120>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-bqwpq4nvz2hs" to finish creating: GET https://rp.j9827200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8f4ae122-44d7-4c73-aff5-38d3b3821a5e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8f4ae122-44d7-4c73-aff5-38d3b3821a5e", "name": "8f4ae122-44d7-4c73-aff5-38d3b3821a5e", "status": "Failed", "startTime": "2026-08-25T07:09:27.254475963Z", "endTime": "2026-08-25T07:28:28.768093899Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc0012c2120>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-bqwpq4nvz2hs" to finish creating: GET https://rp.j9827200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8f4ae122-44d7-4c73-aff5-38d3b3821a5e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8f4ae122-44d7-4c73-aff5-38d3b3821a5e", "name": "8f4ae122-44d7-4c73-aff5-38d3b3821a5e", "status": "Failed", "startTime": "2026-08-25T07:09:27.254475963Z", "endTime": "2026-08-25T07:28:28.768093899Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 06:04:04 UTC · failed at e2e · test FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc000d44a80>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-5kzvdf4f4k42" to finish creating: GET https://rp.j9827200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9d420474-93d9-441d-8bce-471c7c2b5fbc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9d420474-93d9-441d-8bce-471c7c2b5fbc", "name": "9d420474-93d9-441d-8bce-471c7c2b5fbc", "status": "Failed", "startTime": "2026-08-25T07:09:25.661510376Z", "endTime": "2026-08-25T07:28:28.88206189Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc000d44a80>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-5kzvdf4f4k42" to finish creating: GET https://rp.j9827200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9d420474-93d9-441d-8bce-471c7c2b5fbc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9d420474-93d9-441d-8bce-471c7c2b5fbc", "name": "9d420474-93d9-441d-8bce-471c7c2b5fbc", "status": "Failed", "startTime": "2026-08-25T07:09:25.661510376Z", "endTime": "2026-08-25T07:28:28.88206189Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred query-basic: command failed: exit status 1 06:04:04 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.003s. Expected success, but got an error: <errors.errorString \| 0xc0010b4150>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[07:29:52.744]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T07:29:44.6508715Z);\nlet _endTime = datetime(2026-08-25T07:29:44.6508719Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-c86hfsjdphpw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T07:29:52.7254435Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;8f581084-3054-4981-a028-a6e365a00f11\",\r\n \"activityId\": \"86b66dc3-52a7-43ae-9446-0e6a169d09a6\",\r\n \"subActivityId\": \"a4b98a8a-4b86-4300-8041-f9a580f608a5\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"4759775b-7267-467e-8374-de7d5bbd8452\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;8f581084-3054-4981-a028-a6e365a00f11 ARID=86b66dc3-52a7-43ae-9446-0e6a169d09a6 \u003e GW.Http.CallContext/86b66dc3-52a7-43ae-9446-0e6a169d09a6 \u003e GW.ExecuteQuery/4759775b-7267-467e-8374-de7d5bbd8452 \u003e DN.FE.ExecuteQuery/a4b98a8a-4b86-4300-8041-f9a580f608a5)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[07:29:52.794]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T07:29:44.6508715Z);\nlet _endTime = datetime(2026-08-25T07:29:44.6508719Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-c86hfsjdphpw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T07:29:52.7254435Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;8f581084-3054-4981-a028-a6e365a00f11\",\r\n \"activityId\": \"86b66dc3-52a7-43ae-9446-0e6a169d09a6\",\r\n \"subActivityId\": \"a4b98a8a-4b86-4300-8041-f9a580f608a5\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"4759775b-7267-467e-8374-de7d5bbd8452\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;8f581084-3054-4981-a028-a6e365a00f11 ARID=86b66dc3-52a7-43ae-9446-0e6a169d09a6 \u003e GW.Http.CallContext/86b66dc3-52a7-43ae-9446-0e6a169d09a6 \u003e GW.ExecuteQuery/4759775b-7267-467e-8374-de7d5bbd8452 \u003e DN.FE.ExecuteQuery/a4b98a8a-4b86-4300-8041-f9a580f608a5)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[07:29:56.028]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T07:29:52.8580255Z);\nlet _endTime = datetime(2026-08-25T07:29:52.8580259Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-c86hfsjdphpw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T07:29:56.0095823Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;5a139d03-ea31-4fe0-b213-96a9c3436ffa\",\r\n \"activityId\": \"c6d58314-2577-4cd2-b1d2-521b89903b88\",\r\n \"subActivityId\": \"9ee72512-2d51-4e94-951e-02405f520594\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"2b025d9c-608c-4fc0-932a-9c4ff65e0033\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;5a139d03-ea31-4fe0-b213-96a9c3436ffa ARID=c6d58314-2577-4cd2-b1d2-521b89903b88 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/96a96db8-5e5c-470d-98e7-f595cc8873a0 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/96a96db8-5e5c-470d-98e7-f595cc8873a0 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/2b025d9c-608c-4fc0-932a-9c4ff65e0033 \u003e DN.FE.ExecuteQuery/9ee72512-2d51-4e94-951e-02405f520594)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[07:29:56.254]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T07:29:52.8580255Z);\nlet _endTime = datetime(2026-08-25T07:29:52.8580259Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-c86hfsjdphpw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T07:29:56.0095823Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;5a139d03-ea31-4fe0-b213-96a9c3436ffa\",\r\n \"activityId\": \"c6d58314-2577-4cd2-b1d2-521b89903b88\",\r\n \"subActivityId\": \"9ee72512-2d51-4e94-951e-02405f520594\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"2b025d9c-608c-4fc0-932a-9c4ff65e0033\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;5a139d03-ea31-4fe0-b213-96a9c3436ffa ARID=c6d58314-2577-4cd2-b1d2-521b89903b88 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/96a96db8-5e5c-470d-98e7-f595cc8873a0 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/96a96db8-5e5c-470d-98e7-f595cc8873a0 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/2b025d9c-608c-4fc0-932a-9c4ff65e0033 \u003e DN.FE.ExecuteQuery/9ee72512-2d51-4e94-951e-02405f520594)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[07:30:33.649]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T07:30:25.5485738Z);\nlet _endTime = datetime(2026-08-25T07:30:25.5485742Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-c86hfsjdphpw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T07:30:33.6268189Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;700462f9-237d-4475-9884-f9380159de6d\",\r\n \"activityId\": \"2ccd193c-b19d-4151-99b0-0e1bfdc6d835\",\r\n \"subActivityId\": \"456d8932-f4f5-43fd-b0b8-f104b6e33c9c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f51219a8-503f-4282-b78c-ed0028bde92a\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;700462f9-237d-4475-9884-f9380159de6d ARID=2ccd193c-b19d-4151-99b0-0e1bfdc6d835 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/60f3c18c-8096-456b-8d2d-be70c3a82d69 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/60f3c18c-8096-456b-8d2d-be70c3a82d69 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/f51219a8-503f-4282-b78c-ed0028bde92a \u003e DN.FE.ExecuteQuery/456d8932-f4f5-43fd-b0b8-f104b6e33c9c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[07:30:33.755]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T07:30:25.5485738Z);\nlet _endTime = datetime(2026-08-25T07:30:25.5485742Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-c86hfsjdphpw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T07:30:33.6268189Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;700462f9-237d-4475-9884-f9380159de6d\",\r\n \"activityId\": \"2ccd193c-b19d-4151-99b0-0e1bfdc6d835\",\r\n \"subActivityId\": \"456d8932-f4f5-43fd-b0b8-f104b6e33c9c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f51219a8-503f-4282-b78c-ed0028bde92a\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;700462f9-237d-4475-9884-f9380159de6d ARID=2ccd193c-b19d-4151-99b0-0e1bfdc6d835 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/60f3c18c-8096-456b-8d2d-be70c3a82d69 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/60f3c18c-8096-456b-8d2d-be70c3a82d69 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/f51219a8-503f-4282-b78c-ed0028bde92a \u003e DN.FE.ExecuteQuery/456d8932-f4f5-43fd-b0b8-f104b6e33c9c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.003s. Expected success, but got an error: <errors.errorString \| 0xc0010b4150>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[07:29:52.744]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T07:29:44.6508715Z);\nlet _endTime = datetime(2026-08-25T07:29:44.6508719Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-c86hfsjdphpw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T07:29:52.7254435Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;8f581084-3054-4981-a028-a6e365a00f11\",\r\n \"activityId\": \"86b66dc3-52a7-43ae-9446-0e6a169d09a6\",\r\n \"subActivityId\": \"a4b98a8a-4b86-4300-8041-f9a580f608a5\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"4759775b-7267-467e-8374-de7d5bbd8452\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;8f581084-3054-4981-a028-a6e365a00f11 ARID=86b66dc3-52a7-43ae-9446-0e6a169d09a6 \u003e GW.Http.CallContext/86b66dc3-52a7-43ae-9446-0e6a169d09a6 \u003e GW.ExecuteQuery/4759775b-7267-467e-8374-de7d5bbd8452 \u003e DN.FE.ExecuteQuery/a4b98a8a-4b86-4300-8041-f9a580f608a5)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[07:29:52.794]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T07:29:44.6508715Z);\nlet _endTime = datetime(2026-08-25T07:29:44.6508719Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-c86hfsjdphpw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T07:29:52.7254435Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;8f581084-3054-4981-a028-a6e365a00f11\",\r\n \"activityId\": \"86b66dc3-52a7-43ae-9446-0e6a169d09a6\",\r\n \"subActivityId\": \"a4b98a8a-4b86-4300-8041-f9a580f608a5\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"4759775b-7267-467e-8374-de7d5bbd8452\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;8f581084-3054-4981-a028-a6e365a00f11 ARID=86b66dc3-52a7-43ae-9446-0e6a169d09a6 \u003e GW.Http.CallContext/86b66dc3-52a7-43ae-9446-0e6a169d09a6 \u003e GW.ExecuteQuery/4759775b-7267-467e-8374-de7d5bbd8452 \u003e DN.FE.ExecuteQuery/a4b98a8a-4b86-4300-8041-f9a580f608a5)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[07:29:56.028]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T07:29:52.8580255Z);\nlet _endTime = datetime(2026-08-25T07:29:52.8580259Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-c86hfsjdphpw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T07:29:56.0095823Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;5a139d03-ea31-4fe0-b213-96a9c3436ffa\",\r\n \"activityId\": \"c6d58314-2577-4cd2-b1d2-521b89903b88\",\r\n \"subActivityId\": \"9ee72512-2d51-4e94-951e-02405f520594\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"2b025d9c-608c-4fc0-932a-9c4ff65e0033\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;5a139d03-ea31-4fe0-b213-96a9c3436ffa ARID=c6d58314-2577-4cd2-b1d2-521b89903b88 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/96a96db8-5e5c-470d-98e7-f595cc8873a0 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/96a96db8-5e5c-470d-98e7-f595cc8873a0 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/2b025d9c-608c-4fc0-932a-9c4ff65e0033 \u003e DN.FE.ExecuteQuery/9ee72512-2d51-4e94-951e-02405f520594)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[07:29:56.254]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T07:29:52.8580255Z);\nlet _endTime = datetime(2026-08-25T07:29:52.8580259Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-c86hfsjdphpw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T07:29:56.0095823Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;5a139d03-ea31-4fe0-b213-96a9c3436ffa\",\r\n \"activityId\": \"c6d58314-2577-4cd2-b1d2-521b89903b88\",\r\n \"subActivityId\": \"9ee72512-2d51-4e94-951e-02405f520594\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"2b025d9c-608c-4fc0-932a-9c4ff65e0033\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;5a139d03-ea31-4fe0-b213-96a9c3436ffa ARID=c6d58314-2577-4cd2-b1d2-521b89903b88 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/96a96db8-5e5c-470d-98e7-f595cc8873a0 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/96a96db8-5e5c-470d-98e7-f595cc8873a0 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/2b025d9c-608c-4fc0-932a-9c4ff65e0033 \u003e DN.FE.ExecuteQuery/9ee72512-2d51-4e94-951e-02405f520594)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[07:30:33.649]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T07:30:25.5485738Z);\nlet _endTime = datetime(2026-08-25T07:30:25.5485742Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-c86hfsjdphpw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T07:30:33.6268189Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;700462f9-237d-4475-9884-f9380159de6d\",\r\n \"activityId\": \"2ccd193c-b19d-4151-99b0-0e1bfdc6d835\",\r\n \"subActivityId\": \"456d8932-f4f5-43fd-b0b8-f104b6e33c9c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f51219a8-503f-4282-b78c-ed0028bde92a\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;700462f9-237d-4475-9884-f9380159de6d ARID=2ccd193c-b19d-4151-99b0-0e1bfdc6d835 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/60f3c18c-8096-456b-8d2d-be70c3a82d69 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/60f3c18c-8096-456b-8d2d-be70c3a82d69 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/f51219a8-503f-4282-b78c-ed0028bde92a \u003e DN.FE.ExecuteQuery/456d8932-f4f5-43fd-b0b8-f104b6e33c9c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[07:30:33.755]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T07:30:25.5485738Z);\nlet _endTime = datetime(2026-08-25T07:30:25.5485742Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-c86hfsjdphpw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T07:30:33.6268189Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;700462f9-237d-4475-9884-f9380159de6d\",\r\n \"activityId\": \"2ccd193c-b19d-4151-99b0-0e1bfdc6d835\",\r\n \"subActivityId\": \"456d8932-f4f5-43fd-b0b8-f104b6e33c9c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"f51219a8-503f-4282-b78c-ed0028bde92a\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;700462f9-237d-4475-9884-f9380159de6d ARID=2ccd193c-b19d-4151-99b0-0e1bfdc6d835 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/60f3c18c-8096-456b-8d2d-be70c3a82d69 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/60f3c18c-8096-456b-8d2d-be70c3a82d69 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/f51219a8-503f-4282-b78c-ed0028bde92a \u003e DN.FE.ExecuteQuery/456d8932-f4f5-43fd-b0b8-f104b6e33c9c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 06:04:04 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc000f44480>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-pn894lhtsj9n" to finish creating: GET https://rp.j9827200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7f316fee-27a8-46da-955f-a2e939cbe4e5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7f316fee-27a8-46da-955f-a2e939cbe4e5", "name": "7f316fee-27a8-46da-955f-a2e939cbe4e5", "status": "Failed", "startTime": "2026-08-25T07:10:29.36619422Z", "endTime": "2026-08-25T07:29:38.830703335Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc000f44480>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-pn894lhtsj9n" to finish creating: GET https://rp.j9827200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7f316fee-27a8-46da-955f-a2e939cbe4e5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7f316fee-27a8-46da-955f-a2e939cbe4e5", "name": "7f316fee-27a8-46da-955f-a2e939cbe4e5", "status": "Failed", "startTime": "2026-08-25T07:10:29.36619422Z", "endTime": "2026-08-25T07:29:38.830703335Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 06:04:04 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-08-25T07:21:49Z Ended: 2026-08-25T07:41:50Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9827200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdf3a6r28g6plkambnjikihao8o4b98-s6t4e3w5n3n3f4y", pod="router-b5f575fc7-fx84w", severity="warning" Description: Pod ocm-arohcpci01-2sdf3a6r28g6plkambnjikihao8o4b98-s6t4e3w5n3n3f4y/router-b5f575fc7-fx84w has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-08-25T07:21:49Z Ended: 2026-08-25T07:38:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9827200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdf3pnsd1aqg1g8576apulo6qbhkmek-private-kas", pod="router-bb7884476-jzjwj", severity="warning" Description: Pod ocm-arohcpci01-2sdf3pnsd1aqg1g8576apulo6qbhkmek-private-kas/router-bb7884476-jzjwj has been in a non-ready state for longer than 5 minutes.
05:31:26	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 4e67bc7	e2e	failed	★#6554 (open)	1	e2e (1) query-basic: command failed: exit status 1 05:31:26 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.002s. Expected success, but got an error: <errors.errorString \| 0xc00118a160>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[06:47:25.114]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T06:47:22.1255109Z);\nlet _endTime = datetime(2026-08-25T06:47:22.1255116Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-q2kmlds2kptr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T06:47:25.0967135Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;80b0519e-2406-40eb-b816-cd534228b53c\",\r\n \"activityId\": \"7dd64d1c-c51d-4bda-a768-97e8b5ead7bf\",\r\n \"subActivityId\": \"06e1322c-145e-483e-8739-29dc51f280fb\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"9415089d-480a-4c1c-bb14-7fd15cb269d1\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;80b0519e-2406-40eb-b816-cd534228b53c ARID=7dd64d1c-c51d-4bda-a768-97e8b5ead7bf \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/96364a82-62ee-445d-9b0a-e40687bc35e5 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/96364a82-62ee-445d-9b0a-e40687bc35e5 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/9415089d-480a-4c1c-bb14-7fd15cb269d1 \u003e DN.FE.ExecuteQuery/06e1322c-145e-483e-8739-29dc51f280fb)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[06:47:25.165]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T06:47:22.1255109Z);\nlet _endTime = datetime(2026-08-25T06:47:22.1255116Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-q2kmlds2kptr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T06:47:25.0967135Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;80b0519e-2406-40eb-b816-cd534228b53c\",\r\n \"activityId\": \"7dd64d1c-c51d-4bda-a768-97e8b5ead7bf\",\r\n \"subActivityId\": \"06e1322c-145e-483e-8739-29dc51f280fb\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"9415089d-480a-4c1c-bb14-7fd15cb269d1\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;80b0519e-2406-40eb-b816-cd534228b53c ARID=7dd64d1c-c51d-4bda-a768-97e8b5ead7bf \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/96364a82-62ee-445d-9b0a-e40687bc35e5 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/96364a82-62ee-445d-9b0a-e40687bc35e5 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/9415089d-480a-4c1c-bb14-7fd15cb269d1 \u003e DN.FE.ExecuteQuery/06e1322c-145e-483e-8739-29dc51f280fb)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[06:47:28.557]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T06:47:25.2335043Z);\nlet _endTime = datetime(2026-08-25T06:47:25.2335048Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-q2kmlds2kptr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T06:47:28.5392752Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;598e9d85-cc00-49e4-be80-597291b30526\",\r\n \"activityId\": \"30dd1538-d04d-4237-85d2-65955cf8a03b\",\r\n \"subActivityId\": \"70251134-ef12-4e91-8916-1feac4d77ba1\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"93ec09c6-6033-4863-8dbf-9b9d8c6ce41b\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;598e9d85-cc00-49e4-be80-597291b30526 ARID=30dd1538-d04d-4237-85d2-65955cf8a03b \u003e GW.Http.CallContext/30dd1538-d04d-4237-85d2-65955cf8a03b \u003e GW.ExecuteQuery/93ec09c6-6033-4863-8dbf-9b9d8c6ce41b \u003e DN.FE.ExecuteQuery/70251134-ef12-4e91-8916-1feac4d77ba1)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[06:47:28.606]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T06:47:25.2335043Z);\nlet _endTime = datetime(2026-08-25T06:47:25.2335048Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-q2kmlds2kptr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T06:47:28.5392752Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;598e9d85-cc00-49e4-be80-597291b30526\",\r\n \"activityId\": \"30dd1538-d04d-4237-85d2-65955cf8a03b\",\r\n \"subActivityId\": \"70251134-ef12-4e91-8916-1feac4d77ba1\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"93ec09c6-6033-4863-8dbf-9b9d8c6ce41b\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;598e9d85-cc00-49e4-be80-597291b30526 ARID=30dd1538-d04d-4237-85d2-65955cf8a03b \u003e GW.Http.CallContext/30dd1538-d04d-4237-85d2-65955cf8a03b \u003e GW.ExecuteQuery/93ec09c6-6033-4863-8dbf-9b9d8c6ce41b \u003e DN.FE.ExecuteQuery/70251134-ef12-4e91-8916-1feac4d77ba1)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[06:47:34.946]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T06:47:31.8086106Z);\nlet _endTime = datetime(2026-08-25T06:47:31.8086112Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-q2kmlds2kptr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T06:47:34.9242569Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;8bf820ce-7393-4166-a9e0-9c421ec12c32\",\r\n \"activityId\": \"feb416da-c13a-4ccf-8869-2e872303ce63\",\r\n \"subActivityId\": \"8598484e-c376-4d81-8136-c32b1c69a7db\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"541276a3-69a4-4660-b248-9e8d18f733a7\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;8bf820ce-7393-4166-a9e0-9c421ec12c32 ARID=feb416da-c13a-4ccf-8869-2e872303ce63 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/dea56255-2dda-40d2-87a2-e245e7c83acf \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/dea56255-2dda-40d2-87a2-e245e7c83acf \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/541276a3-69a4-4660-b248-9e8d18f733a7 \u003e DN.FE.ExecuteQuery/8598484e-c376-4d81-8136-c32b1c69a7db)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[06:47:35.049]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T06:47:31.8086106Z);\nlet _endTime = datetime(2026-08-25T06:47:31.8086112Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-q2kmlds2kptr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T06:47:34.9242569Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;8bf820ce-7393-4166-a9e0-9c421ec12c32\",\r\n \"activityId\": \"feb416da-c13a-4ccf-8869-2e872303ce63\",\r\n \"subActivityId\": \"8598484e-c376-4d81-8136-c32b1c69a7db\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"541276a3-69a4-4660-b248-9e8d18f733a7\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;8bf820ce-7393-4166-a9e0-9c421ec12c32 ARID=feb416da-c13a-4ccf-8869-2e872303ce63 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/dea56255-2dda-40d2-87a2-e245e7c83acf \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/dea56255-2dda-40d2-87a2-e245e7c83acf \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/541276a3-69a4-4660-b248-9e8d18f733a7 \u003e DN.FE.ExecuteQuery/8598484e-c376-4d81-8136-c32b1c69a7db)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.002s. Expected success, but got an error: <errors.errorString \| 0xc00118a160>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[06:47:25.114]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T06:47:22.1255109Z);\nlet _endTime = datetime(2026-08-25T06:47:22.1255116Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-q2kmlds2kptr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T06:47:25.0967135Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;80b0519e-2406-40eb-b816-cd534228b53c\",\r\n \"activityId\": \"7dd64d1c-c51d-4bda-a768-97e8b5ead7bf\",\r\n \"subActivityId\": \"06e1322c-145e-483e-8739-29dc51f280fb\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"9415089d-480a-4c1c-bb14-7fd15cb269d1\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;80b0519e-2406-40eb-b816-cd534228b53c ARID=7dd64d1c-c51d-4bda-a768-97e8b5ead7bf \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/96364a82-62ee-445d-9b0a-e40687bc35e5 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/96364a82-62ee-445d-9b0a-e40687bc35e5 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/9415089d-480a-4c1c-bb14-7fd15cb269d1 \u003e DN.FE.ExecuteQuery/06e1322c-145e-483e-8739-29dc51f280fb)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[06:47:25.165]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T06:47:22.1255109Z);\nlet _endTime = datetime(2026-08-25T06:47:22.1255116Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-q2kmlds2kptr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T06:47:25.0967135Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;80b0519e-2406-40eb-b816-cd534228b53c\",\r\n \"activityId\": \"7dd64d1c-c51d-4bda-a768-97e8b5ead7bf\",\r\n \"subActivityId\": \"06e1322c-145e-483e-8739-29dc51f280fb\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"9415089d-480a-4c1c-bb14-7fd15cb269d1\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;80b0519e-2406-40eb-b816-cd534228b53c ARID=7dd64d1c-c51d-4bda-a768-97e8b5ead7bf \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/96364a82-62ee-445d-9b0a-e40687bc35e5 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/96364a82-62ee-445d-9b0a-e40687bc35e5 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/9415089d-480a-4c1c-bb14-7fd15cb269d1 \u003e DN.FE.ExecuteQuery/06e1322c-145e-483e-8739-29dc51f280fb)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[06:47:28.557]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T06:47:25.2335043Z);\nlet _endTime = datetime(2026-08-25T06:47:25.2335048Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-q2kmlds2kptr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T06:47:28.5392752Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;598e9d85-cc00-49e4-be80-597291b30526\",\r\n \"activityId\": \"30dd1538-d04d-4237-85d2-65955cf8a03b\",\r\n \"subActivityId\": \"70251134-ef12-4e91-8916-1feac4d77ba1\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"93ec09c6-6033-4863-8dbf-9b9d8c6ce41b\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;598e9d85-cc00-49e4-be80-597291b30526 ARID=30dd1538-d04d-4237-85d2-65955cf8a03b \u003e GW.Http.CallContext/30dd1538-d04d-4237-85d2-65955cf8a03b \u003e GW.ExecuteQuery/93ec09c6-6033-4863-8dbf-9b9d8c6ce41b \u003e DN.FE.ExecuteQuery/70251134-ef12-4e91-8916-1feac4d77ba1)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[06:47:28.606]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T06:47:25.2335043Z);\nlet _endTime = datetime(2026-08-25T06:47:25.2335048Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-q2kmlds2kptr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T06:47:28.5392752Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;598e9d85-cc00-49e4-be80-597291b30526\",\r\n \"activityId\": \"30dd1538-d04d-4237-85d2-65955cf8a03b\",\r\n \"subActivityId\": \"70251134-ef12-4e91-8916-1feac4d77ba1\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"93ec09c6-6033-4863-8dbf-9b9d8c6ce41b\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;598e9d85-cc00-49e4-be80-597291b30526 ARID=30dd1538-d04d-4237-85d2-65955cf8a03b \u003e GW.Http.CallContext/30dd1538-d04d-4237-85d2-65955cf8a03b \u003e GW.ExecuteQuery/93ec09c6-6033-4863-8dbf-9b9d8c6ce41b \u003e DN.FE.ExecuteQuery/70251134-ef12-4e91-8916-1feac4d77ba1)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[06:47:34.946]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T06:47:31.8086106Z);\nlet _endTime = datetime(2026-08-25T06:47:31.8086112Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-q2kmlds2kptr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T06:47:34.9242569Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;8bf820ce-7393-4166-a9e0-9c421ec12c32\",\r\n \"activityId\": \"feb416da-c13a-4ccf-8869-2e872303ce63\",\r\n \"subActivityId\": \"8598484e-c376-4d81-8136-c32b1c69a7db\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"541276a3-69a4-4660-b248-9e8d18f733a7\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;8bf820ce-7393-4166-a9e0-9c421ec12c32 ARID=feb416da-c13a-4ccf-8869-2e872303ce63 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/dea56255-2dda-40d2-87a2-e245e7c83acf \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/dea56255-2dda-40d2-87a2-e245e7c83acf \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/541276a3-69a4-4660-b248-9e8d18f733a7 \u003e DN.FE.ExecuteQuery/8598484e-c376-4d81-8136-c32b1c69a7db)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[06:47:35.049]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T06:47:31.8086106Z);\nlet _endTime = datetime(2026-08-25T06:47:31.8086112Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-q2kmlds2kptr'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T06:47:34.9242569Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;8bf820ce-7393-4166-a9e0-9c421ec12c32\",\r\n \"activityId\": \"feb416da-c13a-4ccf-8869-2e872303ce63\",\r\n \"subActivityId\": \"8598484e-c376-4d81-8136-c32b1c69a7db\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"541276a3-69a4-4660-b248-9e8d18f733a7\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;8bf820ce-7393-4166-a9e0-9c421ec12c32 ARID=feb416da-c13a-4ccf-8869-2e872303ce63 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/dea56255-2dda-40d2-87a2-e245e7c83acf \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/dea56255-2dda-40d2-87a2-e245e7c83acf \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/541276a3-69a4-4660-b248-9e8d18f733a7 \u003e DN.FE.ExecuteQuery/8598484e-c376-4d81-8136-c32b1c69a7db)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ...
04:39:01	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head a860045 · merge a860045	n/a	passed	#6666 (merged)	0	n/a
04:37:46	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 5d90e24	provision	failed	⚠#6667 (closed)	1	provision (1) ERROR: HTTPSConnectionPool(host='login.microsoftonline.com', port=443): Max retries exceeded with url: /XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/oauth2/v2.0/token (Caused by NameResolutionError("HTTPSConnection(host='login.microsoftonline.com', port=443): Failed 04:37:46 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Maestro.Server/global/mirror-image · suite step graph Show raw failure time=2026-08-25T05:18:44.021Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=global step=mirror-image description="Step mirror-image\n Kind: ImageMirror\n From {<nil> maestro.image.registry <nil>}:{<nil> maestro.image.repository <nil>}@{<nil> maestro.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-08-25T05:19:25.373Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build13.ci.openshift.org registry.build13.ci.openshift.org registry.build13.ci.openshift.org registry.build13.ci.openshift.org registry.build13.ci.openshift.org registry.build13.ci.openshift.org registry.build13.ci.openshift.org registry.build13.ci.openshift.org registry.build13.ci.openshift.org\nFetch pull secret for source registry quay.io from arohcpdev-global KV.\nERROR: HTTPSConnectionPool(host='login.microsoftonline.com', port=443): Max retries exceeded with url: /XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/oauth2/v2.0/token (Caused by NameResolutionError(\"HTTPSConnection(host='login.microsoftonline.com', port=443): Failed to resolve 'login.microsoftonline.com' ([Errno -2] Name or service not known)\"))\n exit status 1"
04:07:14	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 3c81c74	e2e	failed	★#6645 (open)	1	e2e (1) query-basic: command failed: exit status 1 04:07:14 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.009s. Expected success, but got an error: <errors.errorString \| 0xc0017126b0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[05:23:12.259]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T05:23:09.0740987Z);\nlet _endTime = datetime(2026-08-25T05:23:09.0740993Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8mfl5cspbkdx'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T05:23:12.2406274Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;e4713658-d57e-46f4-8a0b-3d523d2d2927\",\r\n \"activityId\": \"67ccf73e-9d69-4840-9d95-7fc3c7426646\",\r\n \"subActivityId\": \"c397c476-48e7-486d-b8c6-5fe149482ad1\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"156f2f9e-5c9d-4528-8d32-c99c4824c78f\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;e4713658-d57e-46f4-8a0b-3d523d2d2927 ARID=67ccf73e-9d69-4840-9d95-7fc3c7426646 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/6b4fc1e1-15ba-4ca3-8fd3-10974ed553ab \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/6b4fc1e1-15ba-4ca3-8fd3-10974ed553ab \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/156f2f9e-5c9d-4528-8d32-c99c4824c78f \u003e DN.FE.ExecuteQuery/c397c476-48e7-486d-b8c6-5fe149482ad1)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[05:23:12.321]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T05:23:09.0740987Z);\nlet _endTime = datetime(2026-08-25T05:23:09.0740993Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8mfl5cspbkdx'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T05:23:12.2406274Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;e4713658-d57e-46f4-8a0b-3d523d2d2927\",\r\n \"activityId\": \"67ccf73e-9d69-4840-9d95-7fc3c7426646\",\r\n \"subActivityId\": \"c397c476-48e7-486d-b8c6-5fe149482ad1\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"156f2f9e-5c9d-4528-8d32-c99c4824c78f\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;e4713658-d57e-46f4-8a0b-3d523d2d2927 ARID=67ccf73e-9d69-4840-9d95-7fc3c7426646 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/6b4fc1e1-15ba-4ca3-8fd3-10974ed553ab \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/6b4fc1e1-15ba-4ca3-8fd3-10974ed553ab \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/156f2f9e-5c9d-4528-8d32-c99c4824c78f \u003e DN.FE.ExecuteQuery/c397c476-48e7-486d-b8c6-5fe149482ad1)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[05:23:15.589]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T05:23:12.3880012Z);\nlet _endTime = datetime(2026-08-25T05:23:12.3880018Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8mfl5cspbkdx'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T05:23:15.5708258Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;6e620926-f28e-4f81-bb34-776e2a4dc71c\",\r\n \"activityId\": \"4ef7f058-fd86-4bd2-9c91-414096eb303d\",\r\n \"subActivityId\": \"1b38b11c-2398-47e8-ab2c-841e6ed710a9\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"1c09422b-21fa-40c4-89c0-61e4139c442e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;6e620926-f28e-4f81-bb34-776e2a4dc71c ARID=4ef7f058-fd86-4bd2-9c91-414096eb303d \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/887aa0ee-f2c6-41e5-bc06-5009a86fede1 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/887aa0ee-f2c6-41e5-bc06-5009a86fede1 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/1c09422b-21fa-40c4-89c0-61e4139c442e \u003e DN.FE.ExecuteQuery/1b38b11c-2398-47e8-ab2c-841e6ed710a9)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[05:23:15.657]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T05:23:12.3880012Z);\nlet _endTime = datetime(2026-08-25T05:23:12.3880018Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8mfl5cspbkdx'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T05:23:15.5708258Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;6e620926-f28e-4f81-bb34-776e2a4dc71c\",\r\n \"activityId\": \"4ef7f058-fd86-4bd2-9c91-414096eb303d\",\r\n \"subActivityId\": \"1b38b11c-2398-47e8-ab2c-841e6ed710a9\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"1c09422b-21fa-40c4-89c0-61e4139c442e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;6e620926-f28e-4f81-bb34-776e2a4dc71c ARID=4ef7f058-fd86-4bd2-9c91-414096eb303d \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/887aa0ee-f2c6-41e5-bc06-5009a86fede1 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/887aa0ee-f2c6-41e5-bc06-5009a86fede1 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/1c09422b-21fa-40c4-89c0-61e4139c442e \u003e DN.FE.ExecuteQuery/1b38b11c-2398-47e8-ab2c-841e6ed710a9)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[05:23:22.264]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T05:23:19.0401870Z);\nlet _endTime = datetime(2026-08-25T05:23:19.0401876Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8mfl5cspbkdx'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T05:23:22.2450735Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;c83e1791-3c16-4d2b-91a6-4658ad24736e\",\r\n \"activityId\": \"dfb66e31-a888-4332-80a1-2c81c1b9fd82\",\r\n \"subActivityId\": \"85c7b110-e789-4805-b1ad-e3b380f7d9dd\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"9c069542-0b38-4e5c-8a35-6a188841ea44\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;c83e1791-3c16-4d2b-91a6-4658ad24736e ARID=dfb66e31-a888-4332-80a1-2c81c1b9fd82 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/606b6bc1-1f5e-4492-8e10-fe308d41c95c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/606b6bc1-1f5e-4492-8e10-fe308d41c95c \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/9c069542-0b38-4e5c-8a35-6a188841ea44 \u003e DN.FE.ExecuteQuery/85c7b110-e789-4805-b1ad-e3b380f7d9dd)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[05:23:22.372]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T05:23:19.0401870Z);\nlet _endTime = datetime(2026-08-25T05:23:19.0401876Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8mfl5cspbkdx'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T05:23:22.2450735Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;c83e1791-3c16-4d2b-91a6-4658ad24736e\",\r\n \"activityId\": \"dfb66e31-a888-4332-80a1-2c81c1b9fd82\",\r\n \"subActivityId\": \"85c7b110-e789-4805-b1ad-e3b380f7d9dd\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"9c069542-0b38-4e5c-8a35-6a188841ea44\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;c83e1791-3c16-4d2b-91a6-4658ad24736e ARID=dfb66e31-a888-4332-80a1-2c81c1b9fd82 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/606b6bc1-1f5e-4492-8e10-fe308d41c95c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/606b6bc1-1f5e-4492-8e10-fe308d41c95c \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/9c069542-0b38-4e5c-8a35-6a188841ea44 \u003e DN.FE.ExecuteQuery/85c7b110-e789-4805-b1ad-e3b380f7d9dd)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.009s. Expected success, but got an error: <errors.errorString \| 0xc0017126b0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[05:23:12.259]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T05:23:09.0740987Z);\nlet _endTime = datetime(2026-08-25T05:23:09.0740993Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8mfl5cspbkdx'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T05:23:12.2406274Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;e4713658-d57e-46f4-8a0b-3d523d2d2927\",\r\n \"activityId\": \"67ccf73e-9d69-4840-9d95-7fc3c7426646\",\r\n \"subActivityId\": \"c397c476-48e7-486d-b8c6-5fe149482ad1\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"156f2f9e-5c9d-4528-8d32-c99c4824c78f\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;e4713658-d57e-46f4-8a0b-3d523d2d2927 ARID=67ccf73e-9d69-4840-9d95-7fc3c7426646 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/6b4fc1e1-15ba-4ca3-8fd3-10974ed553ab \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/6b4fc1e1-15ba-4ca3-8fd3-10974ed553ab \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/156f2f9e-5c9d-4528-8d32-c99c4824c78f \u003e DN.FE.ExecuteQuery/c397c476-48e7-486d-b8c6-5fe149482ad1)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[05:23:12.321]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T05:23:09.0740987Z);\nlet _endTime = datetime(2026-08-25T05:23:09.0740993Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8mfl5cspbkdx'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T05:23:12.2406274Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;e4713658-d57e-46f4-8a0b-3d523d2d2927\",\r\n \"activityId\": \"67ccf73e-9d69-4840-9d95-7fc3c7426646\",\r\n \"subActivityId\": \"c397c476-48e7-486d-b8c6-5fe149482ad1\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"156f2f9e-5c9d-4528-8d32-c99c4824c78f\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;e4713658-d57e-46f4-8a0b-3d523d2d2927 ARID=67ccf73e-9d69-4840-9d95-7fc3c7426646 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/6b4fc1e1-15ba-4ca3-8fd3-10974ed553ab \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/6b4fc1e1-15ba-4ca3-8fd3-10974ed553ab \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/156f2f9e-5c9d-4528-8d32-c99c4824c78f \u003e DN.FE.ExecuteQuery/c397c476-48e7-486d-b8c6-5fe149482ad1)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[05:23:15.589]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T05:23:12.3880012Z);\nlet _endTime = datetime(2026-08-25T05:23:12.3880018Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8mfl5cspbkdx'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T05:23:15.5708258Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;6e620926-f28e-4f81-bb34-776e2a4dc71c\",\r\n \"activityId\": \"4ef7f058-fd86-4bd2-9c91-414096eb303d\",\r\n \"subActivityId\": \"1b38b11c-2398-47e8-ab2c-841e6ed710a9\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"1c09422b-21fa-40c4-89c0-61e4139c442e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;6e620926-f28e-4f81-bb34-776e2a4dc71c ARID=4ef7f058-fd86-4bd2-9c91-414096eb303d \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/887aa0ee-f2c6-41e5-bc06-5009a86fede1 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/887aa0ee-f2c6-41e5-bc06-5009a86fede1 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/1c09422b-21fa-40c4-89c0-61e4139c442e \u003e DN.FE.ExecuteQuery/1b38b11c-2398-47e8-ab2c-841e6ed710a9)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[05:23:15.657]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T05:23:12.3880012Z);\nlet _endTime = datetime(2026-08-25T05:23:12.3880018Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8mfl5cspbkdx'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T05:23:15.5708258Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;6e620926-f28e-4f81-bb34-776e2a4dc71c\",\r\n \"activityId\": \"4ef7f058-fd86-4bd2-9c91-414096eb303d\",\r\n \"subActivityId\": \"1b38b11c-2398-47e8-ab2c-841e6ed710a9\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"1c09422b-21fa-40c4-89c0-61e4139c442e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;6e620926-f28e-4f81-bb34-776e2a4dc71c ARID=4ef7f058-fd86-4bd2-9c91-414096eb303d \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/887aa0ee-f2c6-41e5-bc06-5009a86fede1 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/887aa0ee-f2c6-41e5-bc06-5009a86fede1 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/1c09422b-21fa-40c4-89c0-61e4139c442e \u003e DN.FE.ExecuteQuery/1b38b11c-2398-47e8-ab2c-841e6ed710a9)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[05:23:22.264]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T05:23:19.0401870Z);\nlet _endTime = datetime(2026-08-25T05:23:19.0401876Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8mfl5cspbkdx'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T05:23:22.2450735Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;c83e1791-3c16-4d2b-91a6-4658ad24736e\",\r\n \"activityId\": \"dfb66e31-a888-4332-80a1-2c81c1b9fd82\",\r\n \"subActivityId\": \"85c7b110-e789-4805-b1ad-e3b380f7d9dd\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"9c069542-0b38-4e5c-8a35-6a188841ea44\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;c83e1791-3c16-4d2b-91a6-4658ad24736e ARID=dfb66e31-a888-4332-80a1-2c81c1b9fd82 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/606b6bc1-1f5e-4492-8e10-fe308d41c95c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/606b6bc1-1f5e-4492-8e10-fe308d41c95c \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/9c069542-0b38-4e5c-8a35-6a188841ea44 \u003e DN.FE.ExecuteQuery/85c7b110-e789-4805-b1ad-e3b380f7d9dd)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[05:23:22.372]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T05:23:19.0401870Z);\nlet _endTime = datetime(2026-08-25T05:23:19.0401876Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-8mfl5cspbkdx'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T05:23:22.2450735Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;c83e1791-3c16-4d2b-91a6-4658ad24736e\",\r\n \"activityId\": \"dfb66e31-a888-4332-80a1-2c81c1b9fd82\",\r\n \"subActivityId\": \"85c7b110-e789-4805-b1ad-e3b380f7d9dd\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"9c069542-0b38-4e5c-8a35-6a188841ea44\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;c83e1791-3c16-4d2b-91a6-4658ad24736e ARID=dfb66e31-a888-4332-80a1-2c81c1b9fd82 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/606b6bc1-1f5e-4492-8e10-fe308d41c95c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/606b6bc1-1f5e-4492-8e10-fe308d41c95c \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/9c069542-0b38-4e5c-8a35-6a188841ea44 \u003e DN.FE.ExecuteQuery/85c7b110-e789-4805-b1ad-e3b380f7d9dd)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ...
03:45:56	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 515e34b	alert, e2e	failed	⚠#6583 (open)	6	e2e (5) cluster entered terminal Failed state after role assignment deployment 03:45:56 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.002s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.002s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... query-basic: command failed: exit status 1 03:45:56 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 618.602s. Expected success, but got an error: <errors.errorString \| 0xc00014a160>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[04:54:21.505]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:54:13.5215143Z);\nlet _endTime = datetime(2026-08-25T04:54:13.5215206Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xqbfw8gsxzsz'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:54:21.4861152Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;0391bea9-3d54-4205-8ab5-51cb4fa2457a\",\r\n \"activityId\": \"d67ac573-0f51-4eb2-ac35-09b0ec3653e1\",\r\n \"subActivityId\": \"ab3166e7-5b0b-4155-b8e7-6abe1e54267e\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"825fbb4b-a729-448a-ad52-44fe0cce83c2\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;0391bea9-3d54-4205-8ab5-51cb4fa2457a ARID=d67ac573-0f51-4eb2-ac35-09b0ec3653e1 \u003e GW.Http.CallContext/d67ac573-0f51-4eb2-ac35-09b0ec3653e1 \u003e GW.ExecuteQuery/825fbb4b-a729-448a-ad52-44fe0cce83c2 \u003e DN.FE.ExecuteQuery/ab3166e7-5b0b-4155-b8e7-6abe1e54267e)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[04:54:21.554]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:54:13.5215143Z);\nlet _endTime = datetime(2026-08-25T04:54:13.5215206Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xqbfw8gsxzsz'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:54:21.4861152Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;0391bea9-3d54-4205-8ab5-51cb4fa2457a\",\r\n \"activityId\": \"d67ac573-0f51-4eb2-ac35-09b0ec3653e1\",\r\n \"subActivityId\": \"ab3166e7-5b0b-4155-b8e7-6abe1e54267e\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"825fbb4b-a729-448a-ad52-44fe0cce83c2\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;0391bea9-3d54-4205-8ab5-51cb4fa2457a ARID=d67ac573-0f51-4eb2-ac35-09b0ec3653e1 \u003e GW.Http.CallContext/d67ac573-0f51-4eb2-ac35-09b0ec3653e1 \u003e GW.ExecuteQuery/825fbb4b-a729-448a-ad52-44fe0cce83c2 \u003e DN.FE.ExecuteQuery/ab3166e7-5b0b-4155-b8e7-6abe1e54267e)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[04:54:24.629]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:54:21.6147104Z);\nlet _endTime = datetime(2026-08-25T04:54:21.6147112Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xqbfw8gsxzsz'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:54:24.6091269Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;edf1d2d3-10b4-4f32-a7f1-64c071125b19\",\r\n \"activityId\": \"4d35a4ea-993d-4ed5-bc53-a1db9f2fc22c\",\r\n \"subActivityId\": \"265d8ff4-1ace-44fe-88d7-7084889d640c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"e0ed3c26-1203-4622-839a-062fcf54fd0b\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;edf1d2d3-10b4-4f32-a7f1-64c071125b19 ARID=4d35a4ea-993d-4ed5-bc53-a1db9f2fc22c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f9e96db5-f0f9-40f1-822b-1c1688a3f121 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f9e96db5-f0f9-40f1-822b-1c1688a3f121 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/e0ed3c26-1203-4622-839a-062fcf54fd0b \u003e DN.FE.ExecuteQuery/265d8ff4-1ace-44fe-88d7-7084889d640c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[04:54:24.679]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:54:21.6147104Z);\nlet _endTime = datetime(2026-08-25T04:54:21.6147112Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xqbfw8gsxzsz'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:54:24.6091269Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;edf1d2d3-10b4-4f32-a7f1-64c071125b19\",\r\n \"activityId\": \"4d35a4ea-993d-4ed5-bc53-a1db9f2fc22c\",\r\n \"subActivityId\": \"265d8ff4-1ace-44fe-88d7-7084889d640c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"e0ed3c26-1203-4622-839a-062fcf54fd0b\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;edf1d2d3-10b4-4f32-a7f1-64c071125b19 ARID=4d35a4ea-993d-4ed5-bc53-a1db9f2fc22c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f9e96db5-f0f9-40f1-822b-1c1688a3f121 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f9e96db5-f0f9-40f1-822b-1c1688a3f121 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/e0ed3c26-1203-4622-839a-062fcf54fd0b \u003e DN.FE.ExecuteQuery/265d8ff4-1ace-44fe-88d7-7084889d640c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[04:54:31.005]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:54:28.0321387Z);\nlet _endTime = datetime(2026-08-25T04:54:28.0321392Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xqbfw8gsxzsz'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:54:30.9871955Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;691bc226-47e5-48c8-a838-6ba3be72673c\",\r\n \"activityId\": \"b35e4e3a-fd10-4273-b73d-107f86af424a\",\r\n \"subActivityId\": \"2a561e63-03f6-46d1-8400-8aba70e601b4\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"20ea4571-4aa1-4d90-8612-f4c8adf0d733\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;691bc226-47e5-48c8-a838-6ba3be72673c ARID=b35e4e3a-fd10-4273-b73d-107f86af424a \u003e GW.Http.CallContext/b35e4e3a-fd10-4273-b73d-107f86af424a \u003e GW.ExecuteQuery/20ea4571-4aa1-4d90-8612-f4c8adf0d733 \u003e DN.FE.ExecuteQuery/2a561e63-03f6-46d1-8400-8aba70e601b4)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[04:54:31.097]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:54:28.0321387Z);\nlet _endTime = datetime(2026-08-25T04:54:28.0321392Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xqbfw8gsxzsz'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:54:30.9871955Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;691bc226-47e5-48c8-a838-6ba3be72673c\",\r\n \"activityId\": \"b35e4e3a-fd10-4273-b73d-107f86af424a\",\r\n \"subActivityId\": \"2a561e63-03f6-46d1-8400-8aba70e601b4\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"20ea4571-4aa1-4d90-8612-f4c8adf0d733\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;691bc226-47e5-48c8-a838-6ba3be72673c ARID=b35e4e3a-fd10-4273-b73d-107f86af424a \u003e GW.Http.CallContext/b35e4e3a-fd10-4273-b73d-107f86af424a \u003e GW.ExecuteQuery/20ea4571-4aa1-4d90-8612-f4c8adf0d733 \u003e DN.FE.ExecuteQuery/2a561e63-03f6-46d1-8400-8aba70e601b4)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 618.602s. Expected success, but got an error: <errors.errorString \| 0xc00014a160>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[04:54:21.505]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:54:13.5215143Z);\nlet _endTime = datetime(2026-08-25T04:54:13.5215206Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xqbfw8gsxzsz'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:54:21.4861152Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;0391bea9-3d54-4205-8ab5-51cb4fa2457a\",\r\n \"activityId\": \"d67ac573-0f51-4eb2-ac35-09b0ec3653e1\",\r\n \"subActivityId\": \"ab3166e7-5b0b-4155-b8e7-6abe1e54267e\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"825fbb4b-a729-448a-ad52-44fe0cce83c2\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;0391bea9-3d54-4205-8ab5-51cb4fa2457a ARID=d67ac573-0f51-4eb2-ac35-09b0ec3653e1 \u003e GW.Http.CallContext/d67ac573-0f51-4eb2-ac35-09b0ec3653e1 \u003e GW.ExecuteQuery/825fbb4b-a729-448a-ad52-44fe0cce83c2 \u003e DN.FE.ExecuteQuery/ab3166e7-5b0b-4155-b8e7-6abe1e54267e)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[04:54:21.554]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:54:13.5215143Z);\nlet _endTime = datetime(2026-08-25T04:54:13.5215206Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xqbfw8gsxzsz'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:54:21.4861152Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;0391bea9-3d54-4205-8ab5-51cb4fa2457a\",\r\n \"activityId\": \"d67ac573-0f51-4eb2-ac35-09b0ec3653e1\",\r\n \"subActivityId\": \"ab3166e7-5b0b-4155-b8e7-6abe1e54267e\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"825fbb4b-a729-448a-ad52-44fe0cce83c2\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;0391bea9-3d54-4205-8ab5-51cb4fa2457a ARID=d67ac573-0f51-4eb2-ac35-09b0ec3653e1 \u003e GW.Http.CallContext/d67ac573-0f51-4eb2-ac35-09b0ec3653e1 \u003e GW.ExecuteQuery/825fbb4b-a729-448a-ad52-44fe0cce83c2 \u003e DN.FE.ExecuteQuery/ab3166e7-5b0b-4155-b8e7-6abe1e54267e)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[04:54:24.629]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:54:21.6147104Z);\nlet _endTime = datetime(2026-08-25T04:54:21.6147112Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xqbfw8gsxzsz'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:54:24.6091269Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;edf1d2d3-10b4-4f32-a7f1-64c071125b19\",\r\n \"activityId\": \"4d35a4ea-993d-4ed5-bc53-a1db9f2fc22c\",\r\n \"subActivityId\": \"265d8ff4-1ace-44fe-88d7-7084889d640c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"e0ed3c26-1203-4622-839a-062fcf54fd0b\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;edf1d2d3-10b4-4f32-a7f1-64c071125b19 ARID=4d35a4ea-993d-4ed5-bc53-a1db9f2fc22c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f9e96db5-f0f9-40f1-822b-1c1688a3f121 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f9e96db5-f0f9-40f1-822b-1c1688a3f121 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/e0ed3c26-1203-4622-839a-062fcf54fd0b \u003e DN.FE.ExecuteQuery/265d8ff4-1ace-44fe-88d7-7084889d640c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[04:54:24.679]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:54:21.6147104Z);\nlet _endTime = datetime(2026-08-25T04:54:21.6147112Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xqbfw8gsxzsz'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:54:24.6091269Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;edf1d2d3-10b4-4f32-a7f1-64c071125b19\",\r\n \"activityId\": \"4d35a4ea-993d-4ed5-bc53-a1db9f2fc22c\",\r\n \"subActivityId\": \"265d8ff4-1ace-44fe-88d7-7084889d640c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"e0ed3c26-1203-4622-839a-062fcf54fd0b\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;edf1d2d3-10b4-4f32-a7f1-64c071125b19 ARID=4d35a4ea-993d-4ed5-bc53-a1db9f2fc22c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f9e96db5-f0f9-40f1-822b-1c1688a3f121 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f9e96db5-f0f9-40f1-822b-1c1688a3f121 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/e0ed3c26-1203-4622-839a-062fcf54fd0b \u003e DN.FE.ExecuteQuery/265d8ff4-1ace-44fe-88d7-7084889d640c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[04:54:31.005]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:54:28.0321387Z);\nlet _endTime = datetime(2026-08-25T04:54:28.0321392Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xqbfw8gsxzsz'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:54:30.9871955Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;691bc226-47e5-48c8-a838-6ba3be72673c\",\r\n \"activityId\": \"b35e4e3a-fd10-4273-b73d-107f86af424a\",\r\n \"subActivityId\": \"2a561e63-03f6-46d1-8400-8aba70e601b4\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"20ea4571-4aa1-4d90-8612-f4c8adf0d733\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;691bc226-47e5-48c8-a838-6ba3be72673c ARID=b35e4e3a-fd10-4273-b73d-107f86af424a \u003e GW.Http.CallContext/b35e4e3a-fd10-4273-b73d-107f86af424a \u003e GW.ExecuteQuery/20ea4571-4aa1-4d90-8612-f4c8adf0d733 \u003e DN.FE.ExecuteQuery/2a561e63-03f6-46d1-8400-8aba70e601b4)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[04:54:31.097]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:54:28.0321387Z);\nlet _endTime = datetime(2026-08-25T04:54:28.0321392Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xqbfw8gsxzsz'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:54:30.9871955Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;691bc226-47e5-48c8-a838-6ba3be72673c\",\r\n \"activityId\": \"b35e4e3a-fd10-4273-b73d-107f86af424a\",\r\n \"subActivityId\": \"2a561e63-03f6-46d1-8400-8aba70e601b4\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"20ea4571-4aa1-4d90-8612-f4c8adf0d733\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;691bc226-47e5-48c8-a838-6ba3be72673c ARID=b35e4e3a-fd10-4273-b73d-107f86af424a \u003e GW.Http.CallContext/b35e4e3a-fd10-4273-b73d-107f86af424a \u003e GW.ExecuteQuery/20ea4571-4aa1-4d90-8612-f4c8adf0d733 \u003e DN.FE.ExecuteQuery/2a561e63-03f6-46d1-8400-8aba70e601b4)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... VerifyImageRegistryDisabled failed: wrong type of error: url.Error, Get "<url>": dial tcp: lookup <host> on <dns-server>: read udp <ip>-><ip>: i/o timeout 03:45:56 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:108]: failed to verify image registry is disabled on cluster disabled-image-registry-hcp-cluster Unexpected error: <errors.joinError \| 0xc0012d80d8>: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.m4a3a4d0t8e6l6f.ayq6.j7984128.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices": dial tcp: lookup api.m4a3a4d0t8e6l6f.ayq6.j7984128.hcp.osadev.cloud on 172.30.0.10:53: read udp 10.129.104.78:33481->172.30.0.10:53: i/o timeout VerifyBasicAccess failed: failed to list services: Get "https://api.m4a3a4d0t8e6l6f.ayq6.j7984128.hcp.osadev.cloud:443/api/v1/namespaces/default/services": dial tcp: lookup api.m4a3a4d0t8e6l6f.ayq6.j7984128.hcp.osadev.cloud on 172.30.0.10:53: read udp 10.129.104.78:33481->172.30.0.10:53: i/o timeout VerifyImageRegistryDisabled failed: wrong type of error: url.Error, Get "https://api.m4a3a4d0t8e6l6f.ayq6.j7984128.hcp.osadev.cloud:443/api/v1/namespaces/openshift-image-registry/services/image-registry": dial tcp: lookup api.m4a3a4d0t8e6l6f.ayq6.j7984128.hcp.osadev.cloud on 172.30.0.10:53: read udp 10.129.104.78:33481->172.30.0.10:53: i/o timeout ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:108]: failed to verify image registry is disabled on cluster disabled-image-registry-hcp-cluster Unexpected error: <errors.joinError \| 0xc0012d80d8>: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.m4a3a4d0t8e6l6f.ayq6.j7984128.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices": dial tcp: lookup api.m4a3a4d0t8e6l6f.ayq6.j7984128.hcp.osadev.cloud on 172.30.0.10:53: read udp 10.129.104.78:33481->172.30.0.10:53: i/o timeout VerifyBasicAccess failed: failed to list services: Get "https://api.m4a3a4d0t8e6l6f.ayq6.j7984128.hcp.osadev.cloud:443/api/v1/namespaces/default/services": dial tcp: lookup api.m4a3a4d0t8e6l6f.ayq6.j7984128.hcp.osadev.cloud on 172.30.0.10:53: read udp 10.129.104.78:33481->172.30.0.10:53: i/o timeout VerifyImageRegistryDisabled failed: wrong type of error: url.Error, Get "https://api.m4a3a4d0t8e6l6f.ayq6.j7984128.hcp.osadev.cloud:443/api/v1/namespaces/openshift-image-registry/services/image-registry": dial tcp: lookup api.m4a3a4d0t8e6l6f.ayq6.j7984128.hcp.osadev.cloud on 172.30.0.10:53: read udp 10.129.104.78:33481->172.30.0.10:53: i/o timeout ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 03:45:56 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260630preview and verify the ingress is internal · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc000eaa540>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-cfzqww2rkrtr" to finish creating: GET https://rp.j7984128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2f31d32d-9bbd-49ec-b81e-04190385bf80 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2f31d32d-9bbd-49ec-b81e-04190385bf80", "name": "2f31d32d-9bbd-49ec-b81e-04190385bf80", "status": "Failed", "startTime": "2026-08-25T04:28:47.291958618Z", "endTime": "2026-08-25T04:47:48.392085583Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc000eaa540>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-cfzqww2rkrtr" to finish creating: GET https://rp.j7984128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2f31d32d-9bbd-49ec-b81e-04190385bf80 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2f31d32d-9bbd-49ec-b81e-04190385bf80", "name": "2f31d32d-9bbd-49ec-b81e-04190385bf80", "status": "Failed", "startTime": "2026-08-25T04:28:47.291958618Z", "endTime": "2026-08-25T04:47:48.392085583Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 03:45:56 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc000b20780>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-zmbgntqwnfcl" to finish creating: GET https://rp.j7984128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51812137-4e69-43f4-b5f0-1ba75bc984cb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51812137-4e69-43f4-b5f0-1ba75bc984cb", "name": "51812137-4e69-43f4-b5f0-1ba75bc984cb", "status": "Failed", "startTime": "2026-08-25T04:28:42.421330013Z", "endTime": "2026-08-25T04:47:48.547435521Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <*fmt.wrapError \| 0xc000b20780>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-zmbgntqwnfcl" to finish creating: GET https://rp.j7984128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51812137-4e69-43f4-b5f0-1ba75bc984cb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51812137-4e69-43f4-b5f0-1ba75bc984cb", "name": "51812137-4e69-43f4-b5f0-1ba75bc984cb", "status": "Failed", "startTime": "2026-08-25T04:28:42.421330013Z", "endTime": "2026-08-25T04:47:48.547435521Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 03:45:56 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 7 time(s) (1 unknown, 6 known) Firing 1: State: Resolved Started: 2026-08-25T04:41:18Z Ended: 2026-08-25T05:17:15Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7984128-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdcnke2ne01uj8ljmgfd7q922scclq0-e3y5f4c0x3e4e4z", pod="router-6b8bd96c4f-4slzz", severity="warning" Description: Pod ocm-arohcpci01-2sdcnke2ne01uj8ljmgfd7q922scclq0-e3y5f4c0x3e4e4z/router-6b8bd96c4f-4slzz has been in a non-ready state for longer than 5 minutes.
03:26:17	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 54bb2db	e2e	failed	★#6575 (open)	1	e2e (1) query-basic: command failed: exit status 1 03:26:17 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.001s. Expected success, but got an error: <errors.errorString \| 0xc0000d4c60>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[04:31:41.706]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:31:28.5213712Z);\nlet _endTime = datetime(2026-08-25T04:31:28.5213718Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-2hdph7ndrrzh'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:31:41.6877827Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;94be4808-db76-40dd-aaf6-ce4aad0d6682\",\r\n \"activityId\": \"e9a1ad29-c17e-4936-8b9c-7ea9d7c6cc5f\",\r\n \"subActivityId\": \"cb79ddc5-2e15-4ad7-b4a1-5e8175f94891\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"3c54d0eb-f92a-4fe9-8d67-3ac81bfeeda5\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;94be4808-db76-40dd-aaf6-ce4aad0d6682 ARID=e9a1ad29-c17e-4936-8b9c-7ea9d7c6cc5f \u003e GW.Http.CallContext/e9a1ad29-c17e-4936-8b9c-7ea9d7c6cc5f \u003e GW.ExecuteQuery/3c54d0eb-f92a-4fe9-8d67-3ac81bfeeda5 \u003e DN.FE.ExecuteQuery/cb79ddc5-2e15-4ad7-b4a1-5e8175f94891)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[04:31:41.754]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:31:28.5213712Z);\nlet _endTime = datetime(2026-08-25T04:31:28.5213718Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-2hdph7ndrrzh'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:31:41.6877827Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;94be4808-db76-40dd-aaf6-ce4aad0d6682\",\r\n \"activityId\": \"e9a1ad29-c17e-4936-8b9c-7ea9d7c6cc5f\",\r\n \"subActivityId\": \"cb79ddc5-2e15-4ad7-b4a1-5e8175f94891\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"3c54d0eb-f92a-4fe9-8d67-3ac81bfeeda5\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;94be4808-db76-40dd-aaf6-ce4aad0d6682 ARID=e9a1ad29-c17e-4936-8b9c-7ea9d7c6cc5f \u003e GW.Http.CallContext/e9a1ad29-c17e-4936-8b9c-7ea9d7c6cc5f \u003e GW.ExecuteQuery/3c54d0eb-f92a-4fe9-8d67-3ac81bfeeda5 \u003e DN.FE.ExecuteQuery/cb79ddc5-2e15-4ad7-b4a1-5e8175f94891)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[04:31:44.939]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:31:41.8336429Z);\nlet _endTime = datetime(2026-08-25T04:31:41.8336434Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-2hdph7ndrrzh'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:31:44.9203710Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;de7d3089-a049-4c03-b196-1ccd9497c350\",\r\n \"activityId\": \"666d0df5-25a1-4cfa-9916-e3823a7ba44e\",\r\n \"subActivityId\": \"8ded2343-8896-40df-9a04-f4d05531754b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"57be13cf-b001-4782-a859-f7bffcc24095\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;de7d3089-a049-4c03-b196-1ccd9497c350 ARID=666d0df5-25a1-4cfa-9916-e3823a7ba44e \u003e GW.Http.CallContext/666d0df5-25a1-4cfa-9916-e3823a7ba44e \u003e GW.ExecuteQuery/57be13cf-b001-4782-a859-f7bffcc24095 \u003e DN.FE.ExecuteQuery/8ded2343-8896-40df-9a04-f4d05531754b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[04:31:44.984]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:31:41.8336429Z);\nlet _endTime = datetime(2026-08-25T04:31:41.8336434Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-2hdph7ndrrzh'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:31:44.9203710Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;de7d3089-a049-4c03-b196-1ccd9497c350\",\r\n \"activityId\": \"666d0df5-25a1-4cfa-9916-e3823a7ba44e\",\r\n \"subActivityId\": \"8ded2343-8896-40df-9a04-f4d05531754b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"57be13cf-b001-4782-a859-f7bffcc24095\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;de7d3089-a049-4c03-b196-1ccd9497c350 ARID=666d0df5-25a1-4cfa-9916-e3823a7ba44e \u003e GW.Http.CallContext/666d0df5-25a1-4cfa-9916-e3823a7ba44e \u003e GW.ExecuteQuery/57be13cf-b001-4782-a859-f7bffcc24095 \u003e DN.FE.ExecuteQuery/8ded2343-8896-40df-9a04-f4d05531754b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[04:31:51.226]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:31:48.0313611Z);\nlet _endTime = datetime(2026-08-25T04:31:48.0313617Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-2hdph7ndrrzh'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:31:51.2070563Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;900a470c-95cf-4b56-a536-ffebaed1de7d\",\r\n \"activityId\": \"3c4a33bc-b463-4a6e-9b28-8d52ae986c82\",\r\n \"subActivityId\": \"57b721ec-0423-46f3-86e5-ae1e818f1557\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"bc08fd46-3312-4b38-9391-708c1db998b5\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;900a470c-95cf-4b56-a536-ffebaed1de7d ARID=3c4a33bc-b463-4a6e-9b28-8d52ae986c82 \u003e GW.Http.CallContext/3c4a33bc-b463-4a6e-9b28-8d52ae986c82 \u003e GW.ExecuteQuery/bc08fd46-3312-4b38-9391-708c1db998b5 \u003e DN.FE.ExecuteQuery/57b721ec-0423-46f3-86e5-ae1e818f1557)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[04:31:51.318]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:31:48.0313611Z);\nlet _endTime = datetime(2026-08-25T04:31:48.0313617Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-2hdph7ndrrzh'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:31:51.2070563Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;900a470c-95cf-4b56-a536-ffebaed1de7d\",\r\n \"activityId\": \"3c4a33bc-b463-4a6e-9b28-8d52ae986c82\",\r\n \"subActivityId\": \"57b721ec-0423-46f3-86e5-ae1e818f1557\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"bc08fd46-3312-4b38-9391-708c1db998b5\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;900a470c-95cf-4b56-a536-ffebaed1de7d ARID=3c4a33bc-b463-4a6e-9b28-8d52ae986c82 \u003e GW.Http.CallContext/3c4a33bc-b463-4a6e-9b28-8d52ae986c82 \u003e GW.ExecuteQuery/bc08fd46-3312-4b38-9391-708c1db998b5 \u003e DN.FE.ExecuteQuery/57b721ec-0423-46f3-86e5-ae1e818f1557)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.001s. Expected success, but got an error: <errors.errorString \| 0xc0000d4c60>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[04:31:41.706]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:31:28.5213712Z);\nlet _endTime = datetime(2026-08-25T04:31:28.5213718Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-2hdph7ndrrzh'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:31:41.6877827Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;94be4808-db76-40dd-aaf6-ce4aad0d6682\",\r\n \"activityId\": \"e9a1ad29-c17e-4936-8b9c-7ea9d7c6cc5f\",\r\n \"subActivityId\": \"cb79ddc5-2e15-4ad7-b4a1-5e8175f94891\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"3c54d0eb-f92a-4fe9-8d67-3ac81bfeeda5\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;94be4808-db76-40dd-aaf6-ce4aad0d6682 ARID=e9a1ad29-c17e-4936-8b9c-7ea9d7c6cc5f \u003e GW.Http.CallContext/e9a1ad29-c17e-4936-8b9c-7ea9d7c6cc5f \u003e GW.ExecuteQuery/3c54d0eb-f92a-4fe9-8d67-3ac81bfeeda5 \u003e DN.FE.ExecuteQuery/cb79ddc5-2e15-4ad7-b4a1-5e8175f94891)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[04:31:41.754]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:31:28.5213712Z);\nlet _endTime = datetime(2026-08-25T04:31:28.5213718Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-2hdph7ndrrzh'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:31:41.6877827Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;94be4808-db76-40dd-aaf6-ce4aad0d6682\",\r\n \"activityId\": \"e9a1ad29-c17e-4936-8b9c-7ea9d7c6cc5f\",\r\n \"subActivityId\": \"cb79ddc5-2e15-4ad7-b4a1-5e8175f94891\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"3c54d0eb-f92a-4fe9-8d67-3ac81bfeeda5\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;94be4808-db76-40dd-aaf6-ce4aad0d6682 ARID=e9a1ad29-c17e-4936-8b9c-7ea9d7c6cc5f \u003e GW.Http.CallContext/e9a1ad29-c17e-4936-8b9c-7ea9d7c6cc5f \u003e GW.ExecuteQuery/3c54d0eb-f92a-4fe9-8d67-3ac81bfeeda5 \u003e DN.FE.ExecuteQuery/cb79ddc5-2e15-4ad7-b4a1-5e8175f94891)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[04:31:44.939]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:31:41.8336429Z);\nlet _endTime = datetime(2026-08-25T04:31:41.8336434Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-2hdph7ndrrzh'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:31:44.9203710Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;de7d3089-a049-4c03-b196-1ccd9497c350\",\r\n \"activityId\": \"666d0df5-25a1-4cfa-9916-e3823a7ba44e\",\r\n \"subActivityId\": \"8ded2343-8896-40df-9a04-f4d05531754b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"57be13cf-b001-4782-a859-f7bffcc24095\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;de7d3089-a049-4c03-b196-1ccd9497c350 ARID=666d0df5-25a1-4cfa-9916-e3823a7ba44e \u003e GW.Http.CallContext/666d0df5-25a1-4cfa-9916-e3823a7ba44e \u003e GW.ExecuteQuery/57be13cf-b001-4782-a859-f7bffcc24095 \u003e DN.FE.ExecuteQuery/8ded2343-8896-40df-9a04-f4d05531754b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[04:31:44.984]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:31:41.8336429Z);\nlet _endTime = datetime(2026-08-25T04:31:41.8336434Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-2hdph7ndrrzh'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:31:44.9203710Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;de7d3089-a049-4c03-b196-1ccd9497c350\",\r\n \"activityId\": \"666d0df5-25a1-4cfa-9916-e3823a7ba44e\",\r\n \"subActivityId\": \"8ded2343-8896-40df-9a04-f4d05531754b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"57be13cf-b001-4782-a859-f7bffcc24095\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;de7d3089-a049-4c03-b196-1ccd9497c350 ARID=666d0df5-25a1-4cfa-9916-e3823a7ba44e \u003e GW.Http.CallContext/666d0df5-25a1-4cfa-9916-e3823a7ba44e \u003e GW.ExecuteQuery/57be13cf-b001-4782-a859-f7bffcc24095 \u003e DN.FE.ExecuteQuery/8ded2343-8896-40df-9a04-f4d05531754b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[04:31:51.226]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:31:48.0313611Z);\nlet _endTime = datetime(2026-08-25T04:31:48.0313617Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-2hdph7ndrrzh'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:31:51.2070563Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;900a470c-95cf-4b56-a536-ffebaed1de7d\",\r\n \"activityId\": \"3c4a33bc-b463-4a6e-9b28-8d52ae986c82\",\r\n \"subActivityId\": \"57b721ec-0423-46f3-86e5-ae1e818f1557\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"bc08fd46-3312-4b38-9391-708c1db998b5\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;900a470c-95cf-4b56-a536-ffebaed1de7d ARID=3c4a33bc-b463-4a6e-9b28-8d52ae986c82 \u003e GW.Http.CallContext/3c4a33bc-b463-4a6e-9b28-8d52ae986c82 \u003e GW.ExecuteQuery/bc08fd46-3312-4b38-9391-708c1db998b5 \u003e DN.FE.ExecuteQuery/57b721ec-0423-46f3-86e5-ae1e818f1557)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[04:31:51.318]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:31:48.0313611Z);\nlet _endTime = datetime(2026-08-25T04:31:48.0313617Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-2hdph7ndrrzh'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:31:51.2070563Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;900a470c-95cf-4b56-a536-ffebaed1de7d\",\r\n \"activityId\": \"3c4a33bc-b463-4a6e-9b28-8d52ae986c82\",\r\n \"subActivityId\": \"57b721ec-0423-46f3-86e5-ae1e818f1557\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"bc08fd46-3312-4b38-9391-708c1db998b5\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;900a470c-95cf-4b56-a536-ffebaed1de7d ARID=3c4a33bc-b463-4a6e-9b28-8d52ae986c82 \u003e GW.Http.CallContext/3c4a33bc-b463-4a6e-9b28-8d52ae986c82 \u003e GW.ExecuteQuery/bc08fd46-3312-4b38-9391-708c1db998b5 \u003e DN.FE.ExecuteQuery/57b721ec-0423-46f3-86e5-ae1e818f1557)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ...
03:21:15	pull-ci-Azure-ARO-HCP-main-e2e-parallel head a660b6e	e2e	failed	★#6660 (open)	1	e2e (1) query-basic: command failed: exit status 1 03:21:15 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.000s. Expected success, but got an error: <errors.errorString \| 0xc000c71900>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[04:25:11.894]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:25:08.5118208Z);\nlet _endTime = datetime(2026-08-25T04:25:08.5118214Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bm8wcj6bwd2b'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:25:11.8759716Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;89c18873-3158-47b2-8cc0-5b98350d4b0e\",\r\n \"activityId\": \"71123125-c5bb-4db4-90e8-07275e6d8d7c\",\r\n \"subActivityId\": \"b347ccc1-e421-4a49-b038-2d4854509cad\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"e8d7bd2f-af9b-4360-b688-da8755cdf7c3\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;89c18873-3158-47b2-8cc0-5b98350d4b0e ARID=71123125-c5bb-4db4-90e8-07275e6d8d7c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/9047f797-93e4-4624-9e77-9a442f0115ff \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/9047f797-93e4-4624-9e77-9a442f0115ff \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/e8d7bd2f-af9b-4360-b688-da8755cdf7c3 \u003e DN.FE.ExecuteQuery/b347ccc1-e421-4a49-b038-2d4854509cad)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[04:25:11.947]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:25:08.5118208Z);\nlet _endTime = datetime(2026-08-25T04:25:08.5118214Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bm8wcj6bwd2b'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:25:11.8759716Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;89c18873-3158-47b2-8cc0-5b98350d4b0e\",\r\n \"activityId\": \"71123125-c5bb-4db4-90e8-07275e6d8d7c\",\r\n \"subActivityId\": \"b347ccc1-e421-4a49-b038-2d4854509cad\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"e8d7bd2f-af9b-4360-b688-da8755cdf7c3\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;89c18873-3158-47b2-8cc0-5b98350d4b0e ARID=71123125-c5bb-4db4-90e8-07275e6d8d7c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/9047f797-93e4-4624-9e77-9a442f0115ff \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/9047f797-93e4-4624-9e77-9a442f0115ff \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/e8d7bd2f-af9b-4360-b688-da8755cdf7c3 \u003e DN.FE.ExecuteQuery/b347ccc1-e421-4a49-b038-2d4854509cad)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[04:25:15.243]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:25:12.0169012Z);\nlet _endTime = datetime(2026-08-25T04:25:12.0169016Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bm8wcj6bwd2b'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:25:15.2248416Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a23d2beb-5ad2-4517-8ea1-431e59258275\",\r\n \"activityId\": \"92fd7652-4336-4462-868c-12c3ba469e9d\",\r\n \"subActivityId\": \"81fcb6ea-8be7-40dd-8095-18a8ecb464b2\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"790c1312-33a0-482b-8940-5784463be54f\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a23d2beb-5ad2-4517-8ea1-431e59258275 ARID=92fd7652-4336-4462-868c-12c3ba469e9d \u003e GW.Http.CallContext/92fd7652-4336-4462-868c-12c3ba469e9d \u003e GW.ExecuteQuery/790c1312-33a0-482b-8940-5784463be54f \u003e DN.FE.ExecuteQuery/81fcb6ea-8be7-40dd-8095-18a8ecb464b2)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[04:25:15.308]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:25:12.0169012Z);\nlet _endTime = datetime(2026-08-25T04:25:12.0169016Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bm8wcj6bwd2b'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:25:15.2248416Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a23d2beb-5ad2-4517-8ea1-431e59258275\",\r\n \"activityId\": \"92fd7652-4336-4462-868c-12c3ba469e9d\",\r\n \"subActivityId\": \"81fcb6ea-8be7-40dd-8095-18a8ecb464b2\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"790c1312-33a0-482b-8940-5784463be54f\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a23d2beb-5ad2-4517-8ea1-431e59258275 ARID=92fd7652-4336-4462-868c-12c3ba469e9d \u003e GW.Http.CallContext/92fd7652-4336-4462-868c-12c3ba469e9d \u003e GW.ExecuteQuery/790c1312-33a0-482b-8940-5784463be54f \u003e DN.FE.ExecuteQuery/81fcb6ea-8be7-40dd-8095-18a8ecb464b2)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[04:25:24.336]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:25:20.0719077Z);\nlet _endTime = datetime(2026-08-25T04:25:20.0719082Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bm8wcj6bwd2b'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:25:24.3156878Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;0d5f87f2-9259-48f2-94c1-ca20f51aec74\",\r\n \"activityId\": \"354f4d7a-e0a3-4245-aead-8d5376f818a3\",\r\n \"subActivityId\": \"47228ea1-a92b-46c8-8e17-cb952ef7f8d3\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"87aae106-3cd3-4084-a623-9299e4fdb101\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;0d5f87f2-9259-48f2-94c1-ca20f51aec74 ARID=354f4d7a-e0a3-4245-aead-8d5376f818a3 \u003e GW.Http.CallContext/354f4d7a-e0a3-4245-aead-8d5376f818a3 \u003e GW.ExecuteQuery/87aae106-3cd3-4084-a623-9299e4fdb101 \u003e DN.FE.ExecuteQuery/47228ea1-a92b-46c8-8e17-cb952ef7f8d3)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[04:25:24.699]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:25:20.0719077Z);\nlet _endTime = datetime(2026-08-25T04:25:20.0719082Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bm8wcj6bwd2b'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:25:24.3156878Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;0d5f87f2-9259-48f2-94c1-ca20f51aec74\",\r\n \"activityId\": \"354f4d7a-e0a3-4245-aead-8d5376f818a3\",\r\n \"subActivityId\": \"47228ea1-a92b-46c8-8e17-cb952ef7f8d3\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"87aae106-3cd3-4084-a623-9299e4fdb101\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;0d5f87f2-9259-48f2-94c1-ca20f51aec74 ARID=354f4d7a-e0a3-4245-aead-8d5376f818a3 \u003e GW.Http.CallContext/354f4d7a-e0a3-4245-aead-8d5376f818a3 \u003e GW.ExecuteQuery/87aae106-3cd3-4084-a623-9299e4fdb101 \u003e DN.FE.ExecuteQuery/47228ea1-a92b-46c8-8e17-cb952ef7f8d3)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.000s. Expected success, but got an error: <errors.errorString \| 0xc000c71900>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[04:25:11.894]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:25:08.5118208Z);\nlet _endTime = datetime(2026-08-25T04:25:08.5118214Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bm8wcj6bwd2b'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:25:11.8759716Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;89c18873-3158-47b2-8cc0-5b98350d4b0e\",\r\n \"activityId\": \"71123125-c5bb-4db4-90e8-07275e6d8d7c\",\r\n \"subActivityId\": \"b347ccc1-e421-4a49-b038-2d4854509cad\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"e8d7bd2f-af9b-4360-b688-da8755cdf7c3\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;89c18873-3158-47b2-8cc0-5b98350d4b0e ARID=71123125-c5bb-4db4-90e8-07275e6d8d7c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/9047f797-93e4-4624-9e77-9a442f0115ff \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/9047f797-93e4-4624-9e77-9a442f0115ff \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/e8d7bd2f-af9b-4360-b688-da8755cdf7c3 \u003e DN.FE.ExecuteQuery/b347ccc1-e421-4a49-b038-2d4854509cad)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[04:25:11.947]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:25:08.5118208Z);\nlet _endTime = datetime(2026-08-25T04:25:08.5118214Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bm8wcj6bwd2b'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:25:11.8759716Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;89c18873-3158-47b2-8cc0-5b98350d4b0e\",\r\n \"activityId\": \"71123125-c5bb-4db4-90e8-07275e6d8d7c\",\r\n \"subActivityId\": \"b347ccc1-e421-4a49-b038-2d4854509cad\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"e8d7bd2f-af9b-4360-b688-da8755cdf7c3\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;89c18873-3158-47b2-8cc0-5b98350d4b0e ARID=71123125-c5bb-4db4-90e8-07275e6d8d7c \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/9047f797-93e4-4624-9e77-9a442f0115ff \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/9047f797-93e4-4624-9e77-9a442f0115ff \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/e8d7bd2f-af9b-4360-b688-da8755cdf7c3 \u003e DN.FE.ExecuteQuery/b347ccc1-e421-4a49-b038-2d4854509cad)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[04:25:15.243]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:25:12.0169012Z);\nlet _endTime = datetime(2026-08-25T04:25:12.0169016Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bm8wcj6bwd2b'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:25:15.2248416Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a23d2beb-5ad2-4517-8ea1-431e59258275\",\r\n \"activityId\": \"92fd7652-4336-4462-868c-12c3ba469e9d\",\r\n \"subActivityId\": \"81fcb6ea-8be7-40dd-8095-18a8ecb464b2\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"790c1312-33a0-482b-8940-5784463be54f\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a23d2beb-5ad2-4517-8ea1-431e59258275 ARID=92fd7652-4336-4462-868c-12c3ba469e9d \u003e GW.Http.CallContext/92fd7652-4336-4462-868c-12c3ba469e9d \u003e GW.ExecuteQuery/790c1312-33a0-482b-8940-5784463be54f \u003e DN.FE.ExecuteQuery/81fcb6ea-8be7-40dd-8095-18a8ecb464b2)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[04:25:15.308]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:25:12.0169012Z);\nlet _endTime = datetime(2026-08-25T04:25:12.0169016Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bm8wcj6bwd2b'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:25:15.2248416Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;a23d2beb-5ad2-4517-8ea1-431e59258275\",\r\n \"activityId\": \"92fd7652-4336-4462-868c-12c3ba469e9d\",\r\n \"subActivityId\": \"81fcb6ea-8be7-40dd-8095-18a8ecb464b2\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"790c1312-33a0-482b-8940-5784463be54f\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;a23d2beb-5ad2-4517-8ea1-431e59258275 ARID=92fd7652-4336-4462-868c-12c3ba469e9d \u003e GW.Http.CallContext/92fd7652-4336-4462-868c-12c3ba469e9d \u003e GW.ExecuteQuery/790c1312-33a0-482b-8940-5784463be54f \u003e DN.FE.ExecuteQuery/81fcb6ea-8be7-40dd-8095-18a8ecb464b2)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[04:25:24.336]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:25:20.0719077Z);\nlet _endTime = datetime(2026-08-25T04:25:20.0719082Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bm8wcj6bwd2b'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:25:24.3156878Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;0d5f87f2-9259-48f2-94c1-ca20f51aec74\",\r\n \"activityId\": \"354f4d7a-e0a3-4245-aead-8d5376f818a3\",\r\n \"subActivityId\": \"47228ea1-a92b-46c8-8e17-cb952ef7f8d3\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"87aae106-3cd3-4084-a623-9299e4fdb101\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;0d5f87f2-9259-48f2-94c1-ca20f51aec74 ARID=354f4d7a-e0a3-4245-aead-8d5376f818a3 \u003e GW.Http.CallContext/354f4d7a-e0a3-4245-aead-8d5376f818a3 \u003e GW.ExecuteQuery/87aae106-3cd3-4084-a623-9299e4fdb101 \u003e DN.FE.ExecuteQuery/47228ea1-a92b-46c8-8e17-cb952ef7f8d3)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[04:25:24.699]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T04:25:20.0719077Z);\nlet _endTime = datetime(2026-08-25T04:25:20.0719082Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-bm8wcj6bwd2b'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T04:25:24.3156878Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;0d5f87f2-9259-48f2-94c1-ca20f51aec74\",\r\n \"activityId\": \"354f4d7a-e0a3-4245-aead-8d5376f818a3\",\r\n \"subActivityId\": \"47228ea1-a92b-46c8-8e17-cb952ef7f8d3\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"87aae106-3cd3-4084-a623-9299e4fdb101\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;0d5f87f2-9259-48f2-94c1-ca20f51aec74 ARID=354f4d7a-e0a3-4245-aead-8d5376f818a3 \u003e GW.Http.CallContext/354f4d7a-e0a3-4245-aead-8d5376f818a3 \u003e GW.ExecuteQuery/87aae106-3cd3-4084-a623-9299e4fdb101 \u003e DN.FE.ExecuteQuery/47228ea1-a92b-46c8-8e17-cb952ef7f8d3)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ...
02:21:30	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head a860045 · merge a860045	alert, e2e	failed	#6666 (merged)	9	e2e (8) ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link <url> 02:21:30 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:137]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc00152ad60>: failed to create customer-infra: failed waiting for deployment "customer-infra-np-version-upgrade-cluster-rlhmvt-xv6clc" in resourcegroup="rg-np-version-upgrade-rlhmvt-dxdbz4whmbf6" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-version-upgrade-rlhmvt-dxdbz4whmbf6/providers/Microsoft.Resources/deployments/customer-infra-np-version-upgrade-cluster-rlhmvt-xv6clc/operationStatuses/08584139797153791389 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"error\": {\r\n \"code\": \"VaultAlreadyExists\",\r\n \"message\": \"The vault name 'cust-kv-u3utnosfhzsyu' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:137]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc00152ad60>: failed to create customer-infra: failed waiting for deployment "customer-infra-np-version-upgrade-cluster-rlhmvt-xv6clc" in resourcegroup="rg-np-version-upgrade-rlhmvt-dxdbz4whmbf6" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-version-upgrade-rlhmvt-dxdbz4whmbf6/providers/Microsoft.Resources/deployments/customer-infra-np-version-upgrade-cluster-rlhmvt-xv6clc/operationStatuses/08584139797153791389 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"error\": {\r\n \"code\": \"VaultAlreadyExists\",\r\n \"message\": \"The vault name 'cust-kv-u3utnosfhzsyu' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link <url> 02:21:30 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:137]: failed to create customer resources for y-stream upgrade cluster "cp-ystream-upgrade-4-22-wtgj75" Unexpected error: <fmt.wrapError \| 0xc00152e100>: failed to create customer-infra: failed waiting for deployment "customer-infra-cp-ystream-upgrade-4-22-wtgj75-5swmlf" in resourcegroup="rg-cp-ystream-upgrade-4-22-p5qqvrtstbvw" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cp-ystream-upgrade-4-22-p5qqvrtstbvw/providers/Microsoft.Resources/deployments/customer-infra-cp-ystream-upgrade-4-22-wtgj75-5swmlf/operationStatuses/08584139797161101429 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"error\": {\r\n \"code\": \"VaultAlreadyExists\",\r\n \"message\": \"The vault name 'cust-kv-vcuqi2ogq2kwk' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:137]: failed to create customer resources for y-stream upgrade cluster "cp-ystream-upgrade-4-22-wtgj75" Unexpected error: <fmt.wrapError \| 0xc00152e100>: failed to create customer-infra: failed waiting for deployment "customer-infra-cp-ystream-upgrade-4-22-wtgj75-5swmlf" in resourcegroup="rg-cp-ystream-upgrade-4-22-p5qqvrtstbvw" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cp-ystream-upgrade-4-22-p5qqvrtstbvw/providers/Microsoft.Resources/deployments/customer-infra-cp-ystream-upgrade-4-22-wtgj75-5swmlf/operationStatuses/08584139797161101429 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"error\": {\r\n \"code\": \"VaultAlreadyExists\",\r\n \"message\": \"The vault name 'cust-kv-vcuqi2ogq2kwk' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link <url> 02:21:30 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:78]: failed to create customer resources for external auth list test Unexpected error: <fmt.wrapError \| 0xc00184e0e0>: failed to create customer-infra: failed waiting for deployment "customer-infra-ea-listd9kcp5-zjrzlw" in resourcegroup="ea-list-db5549k8k7r7" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/ea-list-db5549k8k7r7/providers/Microsoft.Resources/deployments/customer-infra-ea-listd9kcp5-zjrzlw/operationStatuses/08584139797158020904 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"error\": {\r\n \"code\": \"VaultAlreadyExists\",\r\n \"message\": \"The vault name 'cust-kv-yygnj2pbjetam' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:78]: failed to create customer resources for external auth list test Unexpected error: <fmt.wrapError \| 0xc00184e0e0>: failed to create customer-infra: failed waiting for deployment "customer-infra-ea-listd9kcp5-zjrzlw" in resourcegroup="ea-list-db5549k8k7r7" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/ea-list-db5549k8k7r7/providers/Microsoft.Resources/deployments/customer-infra-ea-listd9kcp5-zjrzlw/operationStatuses/08584139797158020904 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"error\": {\r\n \"code\": \"VaultAlreadyExists\",\r\n \"message\": \"The vault name 'cust-kv-yygnj2pbjetam' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link <url> 02:21:30 UTC · failed at e2e · test SRE should return 409 when boot diagnostics is disabled on a VM · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:407]: failed to create customer resources for boot diagnostics cluster Unexpected error: <fmt.wrapError \| 0xc00124a120>: failed to create customer-infra: failed waiting for deployment "customer-infra-sre-hcp-cluster-bootdiag-wshfnj" in resourcegroup="admin-api-serialconsole-bootdiag-vlm24jhwfxfh" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/admin-api-serialconsole-bootdiag-vlm24jhwfxfh/providers/Microsoft.Resources/deployments/customer-infra-sre-hcp-cluster-bootdiag-wshfnj/operationStatuses/08584139797170845816 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"error\": {\r\n \"code\": \"VaultAlreadyExists\",\r\n \"message\": \"The vault name 'cust-kv-w3yf2qmes5oby' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:407]: failed to create customer resources for boot diagnostics cluster Unexpected error: <fmt.wrapError \| 0xc00124a120>: failed to create customer-infra: failed waiting for deployment "customer-infra-sre-hcp-cluster-bootdiag-wshfnj" in resourcegroup="admin-api-serialconsole-bootdiag-vlm24jhwfxfh" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/admin-api-serialconsole-bootdiag-vlm24jhwfxfh/providers/Microsoft.Resources/deployments/customer-infra-sre-hcp-cluster-bootdiag-wshfnj/operationStatuses/08584139797170845816 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"error\": {\r\n \"code\": \"VaultAlreadyExists\",\r\n \"message\": \"The vault name 'cust-kv-w3yf2qmes5oby' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred cluster entered terminal Failed state after role assignment deployment 02:21:30 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.002s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.002s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link <url> 02:21:30 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:104]: failed to create customer resources for pull secret cluster Unexpected error: <fmt.wrapError \| 0xc00011d8c0>: failed to create customer-infra: failed waiting for deployment "customer-infra-pullsecret-hcp-cluster-q9hglk" in resourcegroup="pullsecret-test-5kr2hsv6hcfr" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/pullsecret-test-5kr2hsv6hcfr/providers/Microsoft.Resources/deployments/customer-infra-pullsecret-hcp-cluster-q9hglk/operationStatuses/08584139797167198377 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"error\": {\r\n \"code\": \"VaultAlreadyExists\",\r\n \"message\": \"The vault name 'cust-kv-je4kwf4e6h44k' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:104]: failed to create customer resources for pull secret cluster Unexpected error: <fmt.wrapError \| 0xc00011d8c0>: failed to create customer-infra: failed waiting for deployment "customer-infra-pullsecret-hcp-cluster-q9hglk" in resourcegroup="pullsecret-test-5kr2hsv6hcfr" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/pullsecret-test-5kr2hsv6hcfr/providers/Microsoft.Resources/deployments/customer-infra-pullsecret-hcp-cluster-q9hglk/operationStatuses/08584139797167198377 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"error\": {\r\n \"code\": \"VaultAlreadyExists\",\r\n \"message\": \"The vault name 'cust-kv-je4kwf4e6h44k' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 02:21:30 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--rq5v9mrlc9n8/cluster-hs-pre-6fnkjh should provision Unexpected error: <fmt.wrapError \| 0xc000f7ea60>: failed to create HCP cluster cluster-hs-pre-6fnkjh: failed waiting for cluster="cluster-hs-pre-6fnkjh" in resourcegroup="rg-hs-pre--rq5v9mrlc9n8" to finish creating: GET https://rp.j4913920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4d26e415-5eaa-496f-8467-9de6cbee0e6f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4d26e415-5eaa-496f-8467-9de6cbee0e6f", "name": "4d26e415-5eaa-496f-8467-9de6cbee0e6f", "status": "Failed", "startTime": "2026-08-25T03:07:08.995601074Z", "endTime": "2026-08-25T03:26:17.111880002Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--rq5v9mrlc9n8/cluster-hs-pre-6fnkjh should provision Unexpected error: <fmt.wrapError \| 0xc000f7ea60>: failed to create HCP cluster cluster-hs-pre-6fnkjh: failed waiting for cluster="cluster-hs-pre-6fnkjh" in resourcegroup="rg-hs-pre--rq5v9mrlc9n8" to finish creating: GET https://rp.j4913920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4d26e415-5eaa-496f-8467-9de6cbee0e6f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4d26e415-5eaa-496f-8467-9de6cbee0e6f", "name": "4d26e415-5eaa-496f-8467-9de6cbee0e6f", "status": "Failed", "startTime": "2026-08-25T03:07:08.995601074Z", "endTime": "2026-08-25T03:26:17.111880002Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link <url> 02:21:30 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:82]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc000839700>: failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-np-labels-taints-qdpn79" in resourcegroup="rg-np-labels-taints-5t4klzqbfs75" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-labels-taints-5t4klzqbfs75/providers/Microsoft.Resources/deployments/customer-infra-cluster-np-labels-taints-qdpn79/operationStatuses/08584139797158851285 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"error\": {\r\n \"code\": \"VaultAlreadyExists\",\r\n \"message\": \"The vault name 'cust-kv-eajlprfd56pyu' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:82]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc000839700>: failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-np-labels-taints-qdpn79" in resourcegroup="rg-np-labels-taints-5t4klzqbfs75" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-labels-taints-5t4klzqbfs75/providers/Microsoft.Resources/deployments/customer-infra-cluster-np-labels-taints-qdpn79/operationStatuses/08584139797158851285 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"error\": {\r\n \"code\": \"VaultAlreadyExists\",\r\n \"message\": \"The vault name 'cust-kv-eajlprfd56pyu' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 02:21:30 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-25T03:20:32Z Ended: 2026-08-25T03:39:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j4913920-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sdbhnin5nlje0kom9tdeu4uptgmg6tm-u0k3n4i5m6h1a0q", pod="router-559bc78df9-qwb7n", severity="warning" Description: Pod ocm-arohcpci01-2sdbhnin5nlje0kom9tdeu4uptgmg6tm-u0k3n4i5m6h1a0q/router-559bc78df9-qwb7n has been in a non-ready state for longer than 5 minutes.
02:08:39	pull-ci-Azure-ARO-HCP-main-e2e-parallel head d5a9f48	alert, e2e	failed	★#6648 (open)	2	e2e (1) query-basic: command failed: exit status 1 02:08:39 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.000s. Expected success, but got an error: <errors.errorString \| 0xc0011bc0e0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[03:21:55.145]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T03:21:50.6783060Z);\nlet _endTime = datetime(2026-08-25T03:21:50.6783064Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-9f2h9xs6hlg2'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T03:21:55.1228509Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;1dd61c9f-5e11-4510-b677-6a52785b7a80\",\r\n \"activityId\": \"a533b28a-fda6-4117-9266-cedeb23e7dcb\",\r\n \"subActivityId\": \"9af34922-404d-4aac-a338-76029942029a\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"717b06d7-8fad-4d02-95bc-b64968fec11b\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;1dd61c9f-5e11-4510-b677-6a52785b7a80 ARID=a533b28a-fda6-4117-9266-cedeb23e7dcb \u003e GW.Http.CallContext/a533b28a-fda6-4117-9266-cedeb23e7dcb \u003e GW.ExecuteQuery/717b06d7-8fad-4d02-95bc-b64968fec11b \u003e DN.FE.ExecuteQuery/9af34922-404d-4aac-a338-76029942029a)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[03:21:55.223]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T03:21:50.6783060Z);\nlet _endTime = datetime(2026-08-25T03:21:50.6783064Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-9f2h9xs6hlg2'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T03:21:55.1228509Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;1dd61c9f-5e11-4510-b677-6a52785b7a80\",\r\n \"activityId\": \"a533b28a-fda6-4117-9266-cedeb23e7dcb\",\r\n \"subActivityId\": \"9af34922-404d-4aac-a338-76029942029a\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"717b06d7-8fad-4d02-95bc-b64968fec11b\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;1dd61c9f-5e11-4510-b677-6a52785b7a80 ARID=a533b28a-fda6-4117-9266-cedeb23e7dcb \u003e GW.Http.CallContext/a533b28a-fda6-4117-9266-cedeb23e7dcb \u003e GW.ExecuteQuery/717b06d7-8fad-4d02-95bc-b64968fec11b \u003e DN.FE.ExecuteQuery/9af34922-404d-4aac-a338-76029942029a)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[03:21:59.054]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T03:21:55.2959172Z);\nlet _endTime = datetime(2026-08-25T03:21:55.2959176Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-9f2h9xs6hlg2'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T03:21:59.0220720Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;70efb98e-3fd3-4ed1-8a8f-46228dd97b8a\",\r\n \"activityId\": \"30c06b36-6f95-41ea-b07a-99c2df6725ac\",\r\n \"subActivityId\": \"f0448801-56e1-4549-948a-2ee0610e96e8\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"e176f93c-c918-49b7-a0cc-4942b4706d95\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;70efb98e-3fd3-4ed1-8a8f-46228dd97b8a ARID=30c06b36-6f95-41ea-b07a-99c2df6725ac \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/71bb00b7-67f3-4178-9894-a7f1ac28d455 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/71bb00b7-67f3-4178-9894-a7f1ac28d455 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/e176f93c-c918-49b7-a0cc-4942b4706d95 \u003e DN.FE.ExecuteQuery/f0448801-56e1-4549-948a-2ee0610e96e8)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[03:21:59.140]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T03:21:55.2959172Z);\nlet _endTime = datetime(2026-08-25T03:21:55.2959176Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-9f2h9xs6hlg2'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T03:21:59.0220720Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;70efb98e-3fd3-4ed1-8a8f-46228dd97b8a\",\r\n \"activityId\": \"30c06b36-6f95-41ea-b07a-99c2df6725ac\",\r\n \"subActivityId\": \"f0448801-56e1-4549-948a-2ee0610e96e8\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"e176f93c-c918-49b7-a0cc-4942b4706d95\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;70efb98e-3fd3-4ed1-8a8f-46228dd97b8a ARID=30c06b36-6f95-41ea-b07a-99c2df6725ac \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/71bb00b7-67f3-4178-9894-a7f1ac28d455 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/71bb00b7-67f3-4178-9894-a7f1ac28d455 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/e176f93c-c918-49b7-a0cc-4942b4706d95 \u003e DN.FE.ExecuteQuery/f0448801-56e1-4549-948a-2ee0610e96e8)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[03:22:08.110]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T03:22:03.5199134Z);\nlet _endTime = datetime(2026-08-25T03:22:03.5199138Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-9f2h9xs6hlg2'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T03:22:08.0879976Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;d2623593-01a6-417f-bb81-7935f556c2fb\",\r\n \"activityId\": \"a25a050e-8242-4a91-95a9-86fa711bcc5a\",\r\n \"subActivityId\": \"8e9b47fe-6372-4513-bf48-f4ed0d67e92b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"8cd9d751-12b3-44c4-9789-8e18a838b311\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;d2623593-01a6-417f-bb81-7935f556c2fb ARID=a25a050e-8242-4a91-95a9-86fa711bcc5a \u003e GW.Http.CallContext/a25a050e-8242-4a91-95a9-86fa711bcc5a \u003e GW.ExecuteQuery/8cd9d751-12b3-44c4-9789-8e18a838b311 \u003e DN.FE.ExecuteQuery/8e9b47fe-6372-4513-bf48-f4ed0d67e92b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[03:22:08.617]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T03:22:03.5199134Z);\nlet _endTime = datetime(2026-08-25T03:22:03.5199138Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-9f2h9xs6hlg2'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T03:22:08.0879976Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;d2623593-01a6-417f-bb81-7935f556c2fb\",\r\n \"activityId\": \"a25a050e-8242-4a91-95a9-86fa711bcc5a\",\r\n \"subActivityId\": \"8e9b47fe-6372-4513-bf48-f4ed0d67e92b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"8cd9d751-12b3-44c4-9789-8e18a838b311\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;d2623593-01a6-417f-bb81-7935f556c2fb ARID=a25a050e-8242-4a91-95a9-86fa711bcc5a \u003e GW.Http.CallContext/a25a050e-8242-4a91-95a9-86fa711bcc5a \u003e GW.ExecuteQuery/8cd9d751-12b3-44c4-9789-8e18a838b311 \u003e DN.FE.ExecuteQuery/8e9b47fe-6372-4513-bf48-f4ed0d67e92b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.000s. Expected success, but got an error: <errors.errorString \| 0xc0011bc0e0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[03:21:55.145]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T03:21:50.6783060Z);\nlet _endTime = datetime(2026-08-25T03:21:50.6783064Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-9f2h9xs6hlg2'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T03:21:55.1228509Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;1dd61c9f-5e11-4510-b677-6a52785b7a80\",\r\n \"activityId\": \"a533b28a-fda6-4117-9266-cedeb23e7dcb\",\r\n \"subActivityId\": \"9af34922-404d-4aac-a338-76029942029a\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"717b06d7-8fad-4d02-95bc-b64968fec11b\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;1dd61c9f-5e11-4510-b677-6a52785b7a80 ARID=a533b28a-fda6-4117-9266-cedeb23e7dcb \u003e GW.Http.CallContext/a533b28a-fda6-4117-9266-cedeb23e7dcb \u003e GW.ExecuteQuery/717b06d7-8fad-4d02-95bc-b64968fec11b \u003e DN.FE.ExecuteQuery/9af34922-404d-4aac-a338-76029942029a)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[03:21:55.223]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T03:21:50.6783060Z);\nlet _endTime = datetime(2026-08-25T03:21:50.6783064Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-9f2h9xs6hlg2'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T03:21:55.1228509Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;1dd61c9f-5e11-4510-b677-6a52785b7a80\",\r\n \"activityId\": \"a533b28a-fda6-4117-9266-cedeb23e7dcb\",\r\n \"subActivityId\": \"9af34922-404d-4aac-a338-76029942029a\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"717b06d7-8fad-4d02-95bc-b64968fec11b\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;1dd61c9f-5e11-4510-b677-6a52785b7a80 ARID=a533b28a-fda6-4117-9266-cedeb23e7dcb \u003e GW.Http.CallContext/a533b28a-fda6-4117-9266-cedeb23e7dcb \u003e GW.ExecuteQuery/717b06d7-8fad-4d02-95bc-b64968fec11b \u003e DN.FE.ExecuteQuery/9af34922-404d-4aac-a338-76029942029a)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[03:21:59.054]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T03:21:55.2959172Z);\nlet _endTime = datetime(2026-08-25T03:21:55.2959176Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-9f2h9xs6hlg2'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T03:21:59.0220720Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;70efb98e-3fd3-4ed1-8a8f-46228dd97b8a\",\r\n \"activityId\": \"30c06b36-6f95-41ea-b07a-99c2df6725ac\",\r\n \"subActivityId\": \"f0448801-56e1-4549-948a-2ee0610e96e8\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"e176f93c-c918-49b7-a0cc-4942b4706d95\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;70efb98e-3fd3-4ed1-8a8f-46228dd97b8a ARID=30c06b36-6f95-41ea-b07a-99c2df6725ac \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/71bb00b7-67f3-4178-9894-a7f1ac28d455 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/71bb00b7-67f3-4178-9894-a7f1ac28d455 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/e176f93c-c918-49b7-a0cc-4942b4706d95 \u003e DN.FE.ExecuteQuery/f0448801-56e1-4549-948a-2ee0610e96e8)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[03:21:59.140]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T03:21:55.2959172Z);\nlet _endTime = datetime(2026-08-25T03:21:55.2959176Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-9f2h9xs6hlg2'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T03:21:59.0220720Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;70efb98e-3fd3-4ed1-8a8f-46228dd97b8a\",\r\n \"activityId\": \"30c06b36-6f95-41ea-b07a-99c2df6725ac\",\r\n \"subActivityId\": \"f0448801-56e1-4549-948a-2ee0610e96e8\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"e176f93c-c918-49b7-a0cc-4942b4706d95\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;70efb98e-3fd3-4ed1-8a8f-46228dd97b8a ARID=30c06b36-6f95-41ea-b07a-99c2df6725ac \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/71bb00b7-67f3-4178-9894-a7f1ac28d455 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/71bb00b7-67f3-4178-9894-a7f1ac28d455 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/e176f93c-c918-49b7-a0cc-4942b4706d95 \u003e DN.FE.ExecuteQuery/f0448801-56e1-4549-948a-2ee0610e96e8)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[03:22:08.110]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T03:22:03.5199134Z);\nlet _endTime = datetime(2026-08-25T03:22:03.5199138Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-9f2h9xs6hlg2'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T03:22:08.0879976Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;d2623593-01a6-417f-bb81-7935f556c2fb\",\r\n \"activityId\": \"a25a050e-8242-4a91-95a9-86fa711bcc5a\",\r\n \"subActivityId\": \"8e9b47fe-6372-4513-bf48-f4ed0d67e92b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"8cd9d751-12b3-44c4-9789-8e18a838b311\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;d2623593-01a6-417f-bb81-7935f556c2fb ARID=a25a050e-8242-4a91-95a9-86fa711bcc5a \u003e GW.Http.CallContext/a25a050e-8242-4a91-95a9-86fa711bcc5a \u003e GW.ExecuteQuery/8cd9d751-12b3-44c4-9789-8e18a838b311 \u003e DN.FE.ExecuteQuery/8e9b47fe-6372-4513-bf48-f4ed0d67e92b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[03:22:08.617]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T03:22:03.5199134Z);\nlet _endTime = datetime(2026-08-25T03:22:03.5199138Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-9f2h9xs6hlg2'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T03:22:08.0879976Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;d2623593-01a6-417f-bb81-7935f556c2fb\",\r\n \"activityId\": \"a25a050e-8242-4a91-95a9-86fa711bcc5a\",\r\n \"subActivityId\": \"8e9b47fe-6372-4513-bf48-f4ed0d67e92b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"8cd9d751-12b3-44c4-9789-8e18a838b311\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;d2623593-01a6-417f-bb81-7935f556c2fb ARID=a25a050e-8242-4a91-95a9-86fa711bcc5a \u003e GW.Http.CallContext/a25a050e-8242-4a91-95a9-86fa711bcc5a \u003e GW.ExecuteQuery/8cd9d751-12b3-44c4-9789-8e18a838b311 \u003e DN.FE.ExecuteQuery/8e9b47fe-6372-4513-bf48-f4ed0d67e92b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... alert (1) alert [svc] BackendControllerRetryHotLoop fired 02:08:39 UTC · failed at alert · test [aro-hcp-observability] [svc] alert BackendControllerRetryHotLoop does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-25T03:09:50Z Ended: 2026-08-25T03:32:50Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j8366976-svc", component="backend", name="observemanagedresourcegroup", severity="warning" Description: Backend controller workqueue observemanagedresourcegroup has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work.
02:08:38	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 86bc57f	e2e	failed	★#6658 (open)	2	e2e (2) query-basic: command failed: exit status 1 02:08:38 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.002s. Expected success, but got an error: <errors.errorString \| 0xc00059c490>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[03:18:05.024]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T03:18:01.4708744Z);\nlet _endTime = datetime(2026-08-25T03:18:01.4708751Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lm44zbt5pvpw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T03:18:04.8684708Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;67bf74fc-55e4-44fe-b1ce-3354120e7c43\",\r\n \"activityId\": \"df7eb063-da18-4ce9-b194-404eed18bc06\",\r\n \"subActivityId\": \"ba927633-d0c2-42e7-92bb-ec25d786d33b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"7262baa4-5b98-4685-985c-814b985c2fe1\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;67bf74fc-55e4-44fe-b1ce-3354120e7c43 ARID=df7eb063-da18-4ce9-b194-404eed18bc06 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/380c0153-ebcd-4bae-9813-2119b7da3734 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/380c0153-ebcd-4bae-9813-2119b7da3734 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/7262baa4-5b98-4685-985c-814b985c2fe1 \u003e DN.FE.ExecuteQuery/ba927633-d0c2-42e7-92bb-ec25d786d33b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[03:18:05.528]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T03:18:01.4708744Z);\nlet _endTime = datetime(2026-08-25T03:18:01.4708751Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lm44zbt5pvpw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T03:18:04.8684708Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;67bf74fc-55e4-44fe-b1ce-3354120e7c43\",\r\n \"activityId\": \"df7eb063-da18-4ce9-b194-404eed18bc06\",\r\n \"subActivityId\": \"ba927633-d0c2-42e7-92bb-ec25d786d33b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"7262baa4-5b98-4685-985c-814b985c2fe1\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;67bf74fc-55e4-44fe-b1ce-3354120e7c43 ARID=df7eb063-da18-4ce9-b194-404eed18bc06 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/380c0153-ebcd-4bae-9813-2119b7da3734 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/380c0153-ebcd-4bae-9813-2119b7da3734 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/7262baa4-5b98-4685-985c-814b985c2fe1 \u003e DN.FE.ExecuteQuery/ba927633-d0c2-42e7-92bb-ec25d786d33b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[03:18:08.793]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T03:18:05.5866186Z);\nlet _endTime = datetime(2026-08-25T03:18:05.5866195Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lm44zbt5pvpw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T03:18:08.7716035Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;e8bce747-327d-497b-8794-664b10bd36f2\",\r\n \"activityId\": \"1fbcf01f-9b69-4f3c-8c1a-a38dd75ca84c\",\r\n \"subActivityId\": \"a66f847d-efa4-4dbd-9d36-35f6dcf8cac6\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"17563e2f-0c3a-4f73-a2a6-b81ec16b170d\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;e8bce747-327d-497b-8794-664b10bd36f2 ARID=1fbcf01f-9b69-4f3c-8c1a-a38dd75ca84c \u003e GW.Http.CallContext/1fbcf01f-9b69-4f3c-8c1a-a38dd75ca84c \u003e GW.ExecuteQuery/17563e2f-0c3a-4f73-a2a6-b81ec16b170d \u003e DN.FE.ExecuteQuery/a66f847d-efa4-4dbd-9d36-35f6dcf8cac6)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[03:18:08.843]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T03:18:05.5866186Z);\nlet _endTime = datetime(2026-08-25T03:18:05.5866195Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lm44zbt5pvpw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T03:18:08.7716035Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;e8bce747-327d-497b-8794-664b10bd36f2\",\r\n \"activityId\": \"1fbcf01f-9b69-4f3c-8c1a-a38dd75ca84c\",\r\n \"subActivityId\": \"a66f847d-efa4-4dbd-9d36-35f6dcf8cac6\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"17563e2f-0c3a-4f73-a2a6-b81ec16b170d\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;e8bce747-327d-497b-8794-664b10bd36f2 ARID=1fbcf01f-9b69-4f3c-8c1a-a38dd75ca84c \u003e GW.Http.CallContext/1fbcf01f-9b69-4f3c-8c1a-a38dd75ca84c \u003e GW.ExecuteQuery/17563e2f-0c3a-4f73-a2a6-b81ec16b170d \u003e DN.FE.ExecuteQuery/a66f847d-efa4-4dbd-9d36-35f6dcf8cac6)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[03:18:14.785]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T03:18:11.8591619Z);\nlet _endTime = datetime(2026-08-25T03:18:11.8591624Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lm44zbt5pvpw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T03:18:14.7664174Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;18680eb2-412c-4a27-a7da-c1683ef1b7d3\",\r\n \"activityId\": \"1ab49792-00ec-450e-8935-2f8d1308a67b\",\r\n \"subActivityId\": \"cd0112e8-bbfc-4f60-bbba-6257d1fca5d0\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"a0bd9d76-07f5-4c1e-88af-6e3ce855fa1f\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;18680eb2-412c-4a27-a7da-c1683ef1b7d3 ARID=1ab49792-00ec-450e-8935-2f8d1308a67b \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/09280cdc-64c4-4ec5-8daa-c623f8d1377e \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/09280cdc-64c4-4ec5-8daa-c623f8d1377e \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/a0bd9d76-07f5-4c1e-88af-6e3ce855fa1f \u003e DN.FE.ExecuteQuery/cd0112e8-bbfc-4f60-bbba-6257d1fca5d0)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[03:18:14.879]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T03:18:11.8591619Z);\nlet _endTime = datetime(2026-08-25T03:18:11.8591624Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lm44zbt5pvpw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T03:18:14.7664174Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;18680eb2-412c-4a27-a7da-c1683ef1b7d3\",\r\n \"activityId\": \"1ab49792-00ec-450e-8935-2f8d1308a67b\",\r\n \"subActivityId\": \"cd0112e8-bbfc-4f60-bbba-6257d1fca5d0\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"a0bd9d76-07f5-4c1e-88af-6e3ce855fa1f\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;18680eb2-412c-4a27-a7da-c1683ef1b7d3 ARID=1ab49792-00ec-450e-8935-2f8d1308a67b \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/09280cdc-64c4-4ec5-8daa-c623f8d1377e \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/09280cdc-64c4-4ec5-8daa-c623f8d1377e \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/a0bd9d76-07f5-4c1e-88af-6e3ce855fa1f \u003e DN.FE.ExecuteQuery/cd0112e8-bbfc-4f60-bbba-6257d1fca5d0)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.002s. Expected success, but got an error: <errors.errorString \| 0xc00059c490>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[03:18:05.024]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T03:18:01.4708744Z);\nlet _endTime = datetime(2026-08-25T03:18:01.4708751Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lm44zbt5pvpw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T03:18:04.8684708Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;67bf74fc-55e4-44fe-b1ce-3354120e7c43\",\r\n \"activityId\": \"df7eb063-da18-4ce9-b194-404eed18bc06\",\r\n \"subActivityId\": \"ba927633-d0c2-42e7-92bb-ec25d786d33b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"7262baa4-5b98-4685-985c-814b985c2fe1\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;67bf74fc-55e4-44fe-b1ce-3354120e7c43 ARID=df7eb063-da18-4ce9-b194-404eed18bc06 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/380c0153-ebcd-4bae-9813-2119b7da3734 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/380c0153-ebcd-4bae-9813-2119b7da3734 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/7262baa4-5b98-4685-985c-814b985c2fe1 \u003e DN.FE.ExecuteQuery/ba927633-d0c2-42e7-92bb-ec25d786d33b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[03:18:05.528]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T03:18:01.4708744Z);\nlet _endTime = datetime(2026-08-25T03:18:01.4708751Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lm44zbt5pvpw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T03:18:04.8684708Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;67bf74fc-55e4-44fe-b1ce-3354120e7c43\",\r\n \"activityId\": \"df7eb063-da18-4ce9-b194-404eed18bc06\",\r\n \"subActivityId\": \"ba927633-d0c2-42e7-92bb-ec25d786d33b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"7262baa4-5b98-4685-985c-814b985c2fe1\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;67bf74fc-55e4-44fe-b1ce-3354120e7c43 ARID=df7eb063-da18-4ce9-b194-404eed18bc06 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/380c0153-ebcd-4bae-9813-2119b7da3734 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/380c0153-ebcd-4bae-9813-2119b7da3734 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/7262baa4-5b98-4685-985c-814b985c2fe1 \u003e DN.FE.ExecuteQuery/ba927633-d0c2-42e7-92bb-ec25d786d33b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[03:18:08.793]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T03:18:05.5866186Z);\nlet _endTime = datetime(2026-08-25T03:18:05.5866195Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lm44zbt5pvpw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T03:18:08.7716035Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;e8bce747-327d-497b-8794-664b10bd36f2\",\r\n \"activityId\": \"1fbcf01f-9b69-4f3c-8c1a-a38dd75ca84c\",\r\n \"subActivityId\": \"a66f847d-efa4-4dbd-9d36-35f6dcf8cac6\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"17563e2f-0c3a-4f73-a2a6-b81ec16b170d\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;e8bce747-327d-497b-8794-664b10bd36f2 ARID=1fbcf01f-9b69-4f3c-8c1a-a38dd75ca84c \u003e GW.Http.CallContext/1fbcf01f-9b69-4f3c-8c1a-a38dd75ca84c \u003e GW.ExecuteQuery/17563e2f-0c3a-4f73-a2a6-b81ec16b170d \u003e DN.FE.ExecuteQuery/a66f847d-efa4-4dbd-9d36-35f6dcf8cac6)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[03:18:08.843]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T03:18:05.5866186Z);\nlet _endTime = datetime(2026-08-25T03:18:05.5866195Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lm44zbt5pvpw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T03:18:08.7716035Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;e8bce747-327d-497b-8794-664b10bd36f2\",\r\n \"activityId\": \"1fbcf01f-9b69-4f3c-8c1a-a38dd75ca84c\",\r\n \"subActivityId\": \"a66f847d-efa4-4dbd-9d36-35f6dcf8cac6\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"17563e2f-0c3a-4f73-a2a6-b81ec16b170d\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;e8bce747-327d-497b-8794-664b10bd36f2 ARID=1fbcf01f-9b69-4f3c-8c1a-a38dd75ca84c \u003e GW.Http.CallContext/1fbcf01f-9b69-4f3c-8c1a-a38dd75ca84c \u003e GW.ExecuteQuery/17563e2f-0c3a-4f73-a2a6-b81ec16b170d \u003e DN.FE.ExecuteQuery/a66f847d-efa4-4dbd-9d36-35f6dcf8cac6)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[03:18:14.785]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T03:18:11.8591619Z);\nlet _endTime = datetime(2026-08-25T03:18:11.8591624Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lm44zbt5pvpw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T03:18:14.7664174Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;18680eb2-412c-4a27-a7da-c1683ef1b7d3\",\r\n \"activityId\": \"1ab49792-00ec-450e-8935-2f8d1308a67b\",\r\n \"subActivityId\": \"cd0112e8-bbfc-4f60-bbba-6257d1fca5d0\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"a0bd9d76-07f5-4c1e-88af-6e3ce855fa1f\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;18680eb2-412c-4a27-a7da-c1683ef1b7d3 ARID=1ab49792-00ec-450e-8935-2f8d1308a67b \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/09280cdc-64c4-4ec5-8daa-c623f8d1377e \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/09280cdc-64c4-4ec5-8daa-c623f8d1377e \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/a0bd9d76-07f5-4c1e-88af-6e3ce855fa1f \u003e DN.FE.ExecuteQuery/cd0112e8-bbfc-4f60-bbba-6257d1fca5d0)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[03:18:14.879]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T03:18:11.8591619Z);\nlet _endTime = datetime(2026-08-25T03:18:11.8591624Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-lm44zbt5pvpw'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T03:18:14.7664174Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;18680eb2-412c-4a27-a7da-c1683ef1b7d3\",\r\n \"activityId\": \"1ab49792-00ec-450e-8935-2f8d1308a67b\",\r\n \"subActivityId\": \"cd0112e8-bbfc-4f60-bbba-6257d1fca5d0\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"a0bd9d76-07f5-4c1e-88af-6e3ce855fa1f\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;18680eb2-412c-4a27-a7da-c1683ef1b7d3 ARID=1ab49792-00ec-450e-8935-2f8d1308a67b \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/09280cdc-64c4-4ec5-8daa-c623f8d1377e \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/09280cdc-64c4-4ec5-8daa-c623f8d1377e \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/a0bd9d76-07f5-4c1e-88af-6e3ce855fa1f \u003e DN.FE.ExecuteQuery/cd0112e8-bbfc-4f60-bbba-6257d1fca5d0)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... private ingress should not be reachable from outside the VNet, but connection succeeded 02:08:38 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260630preview and verify the ingress is internal · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:186]: private ingress should not be reachable from outside the VNet, but connection succeeded Expected an error to have occurred. Got: <nil>: ... fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:186]: private ingress should not be reachable from outside the VNet, but connection succeeded Expected an error to have occurred. Got: <nil>: ...
01:48:25	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 3c81c74	e2e	failed	★#6645 (open)	1	e2e (1) query-basic: command failed: exit status 1 01:48:25 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.000s. Expected success, but got an error: <errors.errorString \| 0xc000b44150>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[02:57:13.167]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T02:57:10.1617793Z);\nlet _endTime = datetime(2026-08-25T02:57:10.1617797Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-x67lfjm8dbhl'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T02:57:13.1478158Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;76052f65-2aef-419e-8fea-b6379e3d72a7\",\r\n \"activityId\": \"abc94df6-4dd7-47ea-8276-cda1a81bec1a\",\r\n \"subActivityId\": \"98b66884-faac-4b45-aa7f-137af0a5c09d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"62d02ff5-f34f-441b-90f9-a785f38b3d63\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;76052f65-2aef-419e-8fea-b6379e3d72a7 ARID=abc94df6-4dd7-47ea-8276-cda1a81bec1a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/ca624d7f-ad79-47bc-8157-638140dc10be \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/ca624d7f-ad79-47bc-8157-638140dc10be \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/62d02ff5-f34f-441b-90f9-a785f38b3d63 \u003e DN.FE.ExecuteQuery/98b66884-faac-4b45-aa7f-137af0a5c09d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[02:57:13.217]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T02:57:10.1617793Z);\nlet _endTime = datetime(2026-08-25T02:57:10.1617797Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-x67lfjm8dbhl'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T02:57:13.1478158Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;76052f65-2aef-419e-8fea-b6379e3d72a7\",\r\n \"activityId\": \"abc94df6-4dd7-47ea-8276-cda1a81bec1a\",\r\n \"subActivityId\": \"98b66884-faac-4b45-aa7f-137af0a5c09d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"62d02ff5-f34f-441b-90f9-a785f38b3d63\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;76052f65-2aef-419e-8fea-b6379e3d72a7 ARID=abc94df6-4dd7-47ea-8276-cda1a81bec1a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/ca624d7f-ad79-47bc-8157-638140dc10be \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/ca624d7f-ad79-47bc-8157-638140dc10be \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/62d02ff5-f34f-441b-90f9-a785f38b3d63 \u003e DN.FE.ExecuteQuery/98b66884-faac-4b45-aa7f-137af0a5c09d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[02:57:16.056]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T02:57:13.2830937Z);\nlet _endTime = datetime(2026-08-25T02:57:13.2830941Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-x67lfjm8dbhl'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T02:57:16.0360865Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;5ef0c71c-0b3a-49ca-95fd-0e58a9d1c254\",\r\n \"activityId\": \"1f294821-56f3-486c-844a-fe730b1f9bd6\",\r\n \"subActivityId\": \"fd707038-0134-45f5-9cfe-c1fc2424f14c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"5d8c0304-a8ff-4084-ad97-bb51505ffc4f\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;5ef0c71c-0b3a-49ca-95fd-0e58a9d1c254 ARID=1f294821-56f3-486c-844a-fe730b1f9bd6 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b75f998b-4331-452f-bc8c-8eef2eddc571 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b75f998b-4331-452f-bc8c-8eef2eddc571 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/5d8c0304-a8ff-4084-ad97-bb51505ffc4f \u003e DN.FE.ExecuteQuery/fd707038-0134-45f5-9cfe-c1fc2424f14c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[02:57:16.106]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T02:57:13.2830937Z);\nlet _endTime = datetime(2026-08-25T02:57:13.2830941Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-x67lfjm8dbhl'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T02:57:16.0360865Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;5ef0c71c-0b3a-49ca-95fd-0e58a9d1c254\",\r\n \"activityId\": \"1f294821-56f3-486c-844a-fe730b1f9bd6\",\r\n \"subActivityId\": \"fd707038-0134-45f5-9cfe-c1fc2424f14c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"5d8c0304-a8ff-4084-ad97-bb51505ffc4f\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;5ef0c71c-0b3a-49ca-95fd-0e58a9d1c254 ARID=1f294821-56f3-486c-844a-fe730b1f9bd6 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b75f998b-4331-452f-bc8c-8eef2eddc571 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b75f998b-4331-452f-bc8c-8eef2eddc571 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/5d8c0304-a8ff-4084-ad97-bb51505ffc4f \u003e DN.FE.ExecuteQuery/fd707038-0134-45f5-9cfe-c1fc2424f14c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[02:57:21.868]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T02:57:19.1309905Z);\nlet _endTime = datetime(2026-08-25T02:57:19.1309912Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-x67lfjm8dbhl'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T02:57:21.8451692Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;e9970c55-d97f-4a19-b966-86c28211c9bb\",\r\n \"activityId\": \"d2ea1670-69d4-4889-a9c3-a16935ec60b7\",\r\n \"subActivityId\": \"413c5e56-d83d-4be8-92fc-a41e2c3ddd3b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"d2e955bc-d7f9-4c57-96df-785494e9e685\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;e9970c55-d97f-4a19-b966-86c28211c9bb ARID=d2ea1670-69d4-4889-a9c3-a16935ec60b7 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f00c2e84-e470-4abc-ba5b-53d753930ad7 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f00c2e84-e470-4abc-ba5b-53d753930ad7 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/d2e955bc-d7f9-4c57-96df-785494e9e685 \u003e DN.FE.ExecuteQuery/413c5e56-d83d-4be8-92fc-a41e2c3ddd3b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[02:57:21.972]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T02:57:19.1309905Z);\nlet _endTime = datetime(2026-08-25T02:57:19.1309912Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-x67lfjm8dbhl'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T02:57:21.8451692Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;e9970c55-d97f-4a19-b966-86c28211c9bb\",\r\n \"activityId\": \"d2ea1670-69d4-4889-a9c3-a16935ec60b7\",\r\n \"subActivityId\": \"413c5e56-d83d-4be8-92fc-a41e2c3ddd3b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"d2e955bc-d7f9-4c57-96df-785494e9e685\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;e9970c55-d97f-4a19-b966-86c28211c9bb ARID=d2ea1670-69d4-4889-a9c3-a16935ec60b7 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f00c2e84-e470-4abc-ba5b-53d753930ad7 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f00c2e84-e470-4abc-ba5b-53d753930ad7 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/d2e955bc-d7f9-4c57-96df-785494e9e685 \u003e DN.FE.ExecuteQuery/413c5e56-d83d-4be8-92fc-a41e2c3ddd3b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.000s. Expected success, but got an error: <errors.errorString \| 0xc000b44150>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[02:57:13.167]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T02:57:10.1617793Z);\nlet _endTime = datetime(2026-08-25T02:57:10.1617797Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-x67lfjm8dbhl'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T02:57:13.1478158Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;76052f65-2aef-419e-8fea-b6379e3d72a7\",\r\n \"activityId\": \"abc94df6-4dd7-47ea-8276-cda1a81bec1a\",\r\n \"subActivityId\": \"98b66884-faac-4b45-aa7f-137af0a5c09d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"62d02ff5-f34f-441b-90f9-a785f38b3d63\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;76052f65-2aef-419e-8fea-b6379e3d72a7 ARID=abc94df6-4dd7-47ea-8276-cda1a81bec1a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/ca624d7f-ad79-47bc-8157-638140dc10be \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/ca624d7f-ad79-47bc-8157-638140dc10be \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/62d02ff5-f34f-441b-90f9-a785f38b3d63 \u003e DN.FE.ExecuteQuery/98b66884-faac-4b45-aa7f-137af0a5c09d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[02:57:13.217]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T02:57:10.1617793Z);\nlet _endTime = datetime(2026-08-25T02:57:10.1617797Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-x67lfjm8dbhl'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T02:57:13.1478158Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;76052f65-2aef-419e-8fea-b6379e3d72a7\",\r\n \"activityId\": \"abc94df6-4dd7-47ea-8276-cda1a81bec1a\",\r\n \"subActivityId\": \"98b66884-faac-4b45-aa7f-137af0a5c09d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"62d02ff5-f34f-441b-90f9-a785f38b3d63\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;76052f65-2aef-419e-8fea-b6379e3d72a7 ARID=abc94df6-4dd7-47ea-8276-cda1a81bec1a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/ca624d7f-ad79-47bc-8157-638140dc10be \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/ca624d7f-ad79-47bc-8157-638140dc10be \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/62d02ff5-f34f-441b-90f9-a785f38b3d63 \u003e DN.FE.ExecuteQuery/98b66884-faac-4b45-aa7f-137af0a5c09d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[02:57:16.056]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T02:57:13.2830937Z);\nlet _endTime = datetime(2026-08-25T02:57:13.2830941Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-x67lfjm8dbhl'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T02:57:16.0360865Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;5ef0c71c-0b3a-49ca-95fd-0e58a9d1c254\",\r\n \"activityId\": \"1f294821-56f3-486c-844a-fe730b1f9bd6\",\r\n \"subActivityId\": \"fd707038-0134-45f5-9cfe-c1fc2424f14c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"5d8c0304-a8ff-4084-ad97-bb51505ffc4f\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;5ef0c71c-0b3a-49ca-95fd-0e58a9d1c254 ARID=1f294821-56f3-486c-844a-fe730b1f9bd6 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b75f998b-4331-452f-bc8c-8eef2eddc571 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b75f998b-4331-452f-bc8c-8eef2eddc571 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/5d8c0304-a8ff-4084-ad97-bb51505ffc4f \u003e DN.FE.ExecuteQuery/fd707038-0134-45f5-9cfe-c1fc2424f14c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[02:57:16.106]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T02:57:13.2830937Z);\nlet _endTime = datetime(2026-08-25T02:57:13.2830941Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-x67lfjm8dbhl'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T02:57:16.0360865Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;5ef0c71c-0b3a-49ca-95fd-0e58a9d1c254\",\r\n \"activityId\": \"1f294821-56f3-486c-844a-fe730b1f9bd6\",\r\n \"subActivityId\": \"fd707038-0134-45f5-9cfe-c1fc2424f14c\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"5d8c0304-a8ff-4084-ad97-bb51505ffc4f\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;5ef0c71c-0b3a-49ca-95fd-0e58a9d1c254 ARID=1f294821-56f3-486c-844a-fe730b1f9bd6 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b75f998b-4331-452f-bc8c-8eef2eddc571 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/b75f998b-4331-452f-bc8c-8eef2eddc571 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/5d8c0304-a8ff-4084-ad97-bb51505ffc4f \u003e DN.FE.ExecuteQuery/fd707038-0134-45f5-9cfe-c1fc2424f14c)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[02:57:21.868]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T02:57:19.1309905Z);\nlet _endTime = datetime(2026-08-25T02:57:19.1309912Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-x67lfjm8dbhl'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T02:57:21.8451692Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;e9970c55-d97f-4a19-b966-86c28211c9bb\",\r\n \"activityId\": \"d2ea1670-69d4-4889-a9c3-a16935ec60b7\",\r\n \"subActivityId\": \"413c5e56-d83d-4be8-92fc-a41e2c3ddd3b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"d2e955bc-d7f9-4c57-96df-785494e9e685\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;e9970c55-d97f-4a19-b966-86c28211c9bb ARID=d2ea1670-69d4-4889-a9c3-a16935ec60b7 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f00c2e84-e470-4abc-ba5b-53d753930ad7 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f00c2e84-e470-4abc-ba5b-53d753930ad7 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/d2e955bc-d7f9-4c57-96df-785494e9e685 \u003e DN.FE.ExecuteQuery/413c5e56-d83d-4be8-92fc-a41e2c3ddd3b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[02:57:21.972]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T02:57:19.1309905Z);\nlet _endTime = datetime(2026-08-25T02:57:19.1309912Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-x67lfjm8dbhl'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T02:57:21.8451692Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;e9970c55-d97f-4a19-b966-86c28211c9bb\",\r\n \"activityId\": \"d2ea1670-69d4-4889-a9c3-a16935ec60b7\",\r\n \"subActivityId\": \"413c5e56-d83d-4be8-92fc-a41e2c3ddd3b\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"d2e955bc-d7f9-4c57-96df-785494e9e685\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;e9970c55-d97f-4a19-b966-86c28211c9bb ARID=d2ea1670-69d4-4889-a9c3-a16935ec60b7 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f00c2e84-e470-4abc-ba5b-53d753930ad7 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/f00c2e84-e470-4abc-ba5b-53d753930ad7 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/d2e955bc-d7f9-4c57-96df-785494e9e685 \u003e DN.FE.ExecuteQuery/413c5e56-d83d-4be8-92fc-a41e2c3ddd3b)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ...
01:47:54	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 4e67bc7	e2e	failed	★#6554 (open)	1	e2e (1) query-basic: command failed: exit status 1 01:47:54 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.001s. Expected success, but got an error: <errors.errorString \| 0xc000b8a1a0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[02:46:49.041]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T02:46:44.9847112Z);\nlet _endTime = datetime(2026-08-25T02:46:44.9847117Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6dxkrlzwnlg5'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T02:46:49.0230725Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;c670b93e-8900-4b77-a4d1-df5f562291df\",\r\n \"activityId\": \"10d87903-bbdc-49d7-b182-c7bd286e370d\",\r\n \"subActivityId\": \"5ed51d3f-0d4a-49f1-aac5-f03667ec93a0\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"dbaf8df9-4a6c-4f51-926c-6a1d0b7a7a2e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;c670b93e-8900-4b77-a4d1-df5f562291df ARID=10d87903-bbdc-49d7-b182-c7bd286e370d \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/ec65d60f-4372-4716-b968-285059eae484 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/ec65d60f-4372-4716-b968-285059eae484 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/dbaf8df9-4a6c-4f51-926c-6a1d0b7a7a2e \u003e DN.FE.ExecuteQuery/5ed51d3f-0d4a-49f1-aac5-f03667ec93a0)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[02:46:49.088]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T02:46:44.9847112Z);\nlet _endTime = datetime(2026-08-25T02:46:44.9847117Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6dxkrlzwnlg5'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T02:46:49.0230725Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;c670b93e-8900-4b77-a4d1-df5f562291df\",\r\n \"activityId\": \"10d87903-bbdc-49d7-b182-c7bd286e370d\",\r\n \"subActivityId\": \"5ed51d3f-0d4a-49f1-aac5-f03667ec93a0\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"dbaf8df9-4a6c-4f51-926c-6a1d0b7a7a2e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;c670b93e-8900-4b77-a4d1-df5f562291df ARID=10d87903-bbdc-49d7-b182-c7bd286e370d \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/ec65d60f-4372-4716-b968-285059eae484 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/ec65d60f-4372-4716-b968-285059eae484 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/dbaf8df9-4a6c-4f51-926c-6a1d0b7a7a2e \u003e DN.FE.ExecuteQuery/5ed51d3f-0d4a-49f1-aac5-f03667ec93a0)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[02:46:52.156]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T02:46:49.1536737Z);\nlet _endTime = datetime(2026-08-25T02:46:49.1536742Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6dxkrlzwnlg5'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T02:46:52.1327710Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;5cc7d5ee-afcd-41e8-afbe-777cb651042c\",\r\n \"activityId\": \"66809e6d-c584-4f5e-b6ed-0c23de2db572\",\r\n \"subActivityId\": \"47e81ccf-6d17-42f1-a388-36872a1b72f0\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"09b71b7c-7e90-4aec-ad2b-df776045d2a3\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;5cc7d5ee-afcd-41e8-afbe-777cb651042c ARID=66809e6d-c584-4f5e-b6ed-0c23de2db572 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/78cb5fbb-a4cc-461a-9fcc-8fc8bc845f2a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/78cb5fbb-a4cc-461a-9fcc-8fc8bc845f2a \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/09b71b7c-7e90-4aec-ad2b-df776045d2a3 \u003e DN.FE.ExecuteQuery/47e81ccf-6d17-42f1-a388-36872a1b72f0)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[02:46:52.208]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T02:46:49.1536737Z);\nlet _endTime = datetime(2026-08-25T02:46:49.1536742Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6dxkrlzwnlg5'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T02:46:52.1327710Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;5cc7d5ee-afcd-41e8-afbe-777cb651042c\",\r\n \"activityId\": \"66809e6d-c584-4f5e-b6ed-0c23de2db572\",\r\n \"subActivityId\": \"47e81ccf-6d17-42f1-a388-36872a1b72f0\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"09b71b7c-7e90-4aec-ad2b-df776045d2a3\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;5cc7d5ee-afcd-41e8-afbe-777cb651042c ARID=66809e6d-c584-4f5e-b6ed-0c23de2db572 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/78cb5fbb-a4cc-461a-9fcc-8fc8bc845f2a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/78cb5fbb-a4cc-461a-9fcc-8fc8bc845f2a \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/09b71b7c-7e90-4aec-ad2b-df776045d2a3 \u003e DN.FE.ExecuteQuery/47e81ccf-6d17-42f1-a388-36872a1b72f0)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[02:46:58.829]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T02:46:55.2542072Z);\nlet _endTime = datetime(2026-08-25T02:46:55.2542077Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6dxkrlzwnlg5'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T02:46:58.8067947Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;edeaa02f-046d-43fc-b2ab-62ec451b6908\",\r\n \"activityId\": \"1a1d50bf-a610-43af-ae94-ff7858999b2c\",\r\n \"subActivityId\": \"e3430530-027f-4ee3-a20d-c3539326e24d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"0afc02b0-9367-4ee2-a98c-86038a9bbbcc\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;edeaa02f-046d-43fc-b2ab-62ec451b6908 ARID=1a1d50bf-a610-43af-ae94-ff7858999b2c \u003e GW.Http.CallContext/1a1d50bf-a610-43af-ae94-ff7858999b2c \u003e GW.ExecuteQuery/0afc02b0-9367-4ee2-a98c-86038a9bbbcc \u003e DN.FE.ExecuteQuery/e3430530-027f-4ee3-a20d-c3539326e24d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[02:46:58.936]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T02:46:55.2542072Z);\nlet _endTime = datetime(2026-08-25T02:46:55.2542077Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6dxkrlzwnlg5'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T02:46:58.8067947Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;edeaa02f-046d-43fc-b2ab-62ec451b6908\",\r\n \"activityId\": \"1a1d50bf-a610-43af-ae94-ff7858999b2c\",\r\n \"subActivityId\": \"e3430530-027f-4ee3-a20d-c3539326e24d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"0afc02b0-9367-4ee2-a98c-86038a9bbbcc\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;edeaa02f-046d-43fc-b2ab-62ec451b6908 ARID=1a1d50bf-a610-43af-ae94-ff7858999b2c \u003e GW.Http.CallContext/1a1d50bf-a610-43af-ae94-ff7858999b2c \u003e GW.ExecuteQuery/0afc02b0-9367-4ee2-a98c-86038a9bbbcc \u003e DN.FE.ExecuteQuery/e3430530-027f-4ee3-a20d-c3539326e24d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.001s. Expected success, but got an error: <errors.errorString \| 0xc000b8a1a0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[02:46:49.041]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T02:46:44.9847112Z);\nlet _endTime = datetime(2026-08-25T02:46:44.9847117Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6dxkrlzwnlg5'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T02:46:49.0230725Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;c670b93e-8900-4b77-a4d1-df5f562291df\",\r\n \"activityId\": \"10d87903-bbdc-49d7-b182-c7bd286e370d\",\r\n \"subActivityId\": \"5ed51d3f-0d4a-49f1-aac5-f03667ec93a0\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"dbaf8df9-4a6c-4f51-926c-6a1d0b7a7a2e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;c670b93e-8900-4b77-a4d1-df5f562291df ARID=10d87903-bbdc-49d7-b182-c7bd286e370d \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/ec65d60f-4372-4716-b968-285059eae484 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/ec65d60f-4372-4716-b968-285059eae484 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/dbaf8df9-4a6c-4f51-926c-6a1d0b7a7a2e \u003e DN.FE.ExecuteQuery/5ed51d3f-0d4a-49f1-aac5-f03667ec93a0)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[02:46:49.088]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T02:46:44.9847112Z);\nlet _endTime = datetime(2026-08-25T02:46:44.9847117Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6dxkrlzwnlg5'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T02:46:49.0230725Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;c670b93e-8900-4b77-a4d1-df5f562291df\",\r\n \"activityId\": \"10d87903-bbdc-49d7-b182-c7bd286e370d\",\r\n \"subActivityId\": \"5ed51d3f-0d4a-49f1-aac5-f03667ec93a0\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"dbaf8df9-4a6c-4f51-926c-6a1d0b7a7a2e\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;c670b93e-8900-4b77-a4d1-df5f562291df ARID=10d87903-bbdc-49d7-b182-c7bd286e370d \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/ec65d60f-4372-4716-b968-285059eae484 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/ec65d60f-4372-4716-b968-285059eae484 \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/dbaf8df9-4a6c-4f51-926c-6a1d0b7a7a2e \u003e DN.FE.ExecuteQuery/5ed51d3f-0d4a-49f1-aac5-f03667ec93a0)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[02:46:52.156]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T02:46:49.1536737Z);\nlet _endTime = datetime(2026-08-25T02:46:49.1536742Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6dxkrlzwnlg5'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T02:46:52.1327710Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;5cc7d5ee-afcd-41e8-afbe-777cb651042c\",\r\n \"activityId\": \"66809e6d-c584-4f5e-b6ed-0c23de2db572\",\r\n \"subActivityId\": \"47e81ccf-6d17-42f1-a388-36872a1b72f0\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"09b71b7c-7e90-4aec-ad2b-df776045d2a3\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;5cc7d5ee-afcd-41e8-afbe-777cb651042c ARID=66809e6d-c584-4f5e-b6ed-0c23de2db572 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/78cb5fbb-a4cc-461a-9fcc-8fc8bc845f2a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/78cb5fbb-a4cc-461a-9fcc-8fc8bc845f2a \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/09b71b7c-7e90-4aec-ad2b-df776045d2a3 \u003e DN.FE.ExecuteQuery/47e81ccf-6d17-42f1-a388-36872a1b72f0)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[02:46:52.208]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T02:46:49.1536737Z);\nlet _endTime = datetime(2026-08-25T02:46:49.1536742Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6dxkrlzwnlg5'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T02:46:52.1327710Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;5cc7d5ee-afcd-41e8-afbe-777cb651042c\",\r\n \"activityId\": \"66809e6d-c584-4f5e-b6ed-0c23de2db572\",\r\n \"subActivityId\": \"47e81ccf-6d17-42f1-a388-36872a1b72f0\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"09b71b7c-7e90-4aec-ad2b-df776045d2a3\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;5cc7d5ee-afcd-41e8-afbe-777cb651042c ARID=66809e6d-c584-4f5e-b6ed-0c23de2db572 \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/78cb5fbb-a4cc-461a-9fcc-8fc8bc845f2a \u003e KD.Query.Client.ExecuteQueryAsKustoDataStream/78cb5fbb-a4cc-461a-9fcc-8fc8bc845f2a \u003e P.Grpc.Service.ExecuteQueryAsKustoDataStream2/09b71b7c-7e90-4aec-ad2b-df776045d2a3 \u003e DN.FE.ExecuteQuery/47e81ccf-6d17-42f1-a388-36872a1b72f0)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[02:46:58.829]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T02:46:55.2542072Z);\nlet _endTime = datetime(2026-08-25T02:46:55.2542077Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6dxkrlzwnlg5'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T02:46:58.8067947Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;edeaa02f-046d-43fc-b2ab-62ec451b6908\",\r\n \"activityId\": \"1a1d50bf-a610-43af-ae94-ff7858999b2c\",\r\n \"subActivityId\": \"e3430530-027f-4ee3-a20d-c3539326e24d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"0afc02b0-9367-4ee2-a98c-86038a9bbbcc\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;edeaa02f-046d-43fc-b2ab-62ec451b6908 ARID=1a1d50bf-a610-43af-ae94-ff7858999b2c \u003e GW.Http.CallContext/1a1d50bf-a610-43af-ae94-ff7858999b2c \u003e GW.ExecuteQuery/0afc02b0-9367-4ee2-a98c-86038a9bbbcc \u003e DN.FE.ExecuteQuery/e3430530-027f-4ee3-a20d-c3539326e24d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[02:46:58.936]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T02:46:55.2542072Z);\nlet _endTime = datetime(2026-08-25T02:46:55.2542077Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-6dxkrlzwnlg5'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T02:46:58.8067947Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;edeaa02f-046d-43fc-b2ab-62ec451b6908\",\r\n \"activityId\": \"1a1d50bf-a610-43af-ae94-ff7858999b2c\",\r\n \"subActivityId\": \"e3430530-027f-4ee3-a20d-c3539326e24d\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"0afc02b0-9367-4ee2-a98c-86038a9bbbcc\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;edeaa02f-046d-43fc-b2ab-62ec451b6908 ARID=1a1d50bf-a610-43af-ae94-ff7858999b2c \u003e GW.Http.CallContext/1a1d50bf-a610-43af-ae94-ff7858999b2c \u003e GW.ExecuteQuery/0afc02b0-9367-4ee2-a98c-86038a9bbbcc \u003e DN.FE.ExecuteQuery/e3430530-027f-4ee3-a20d-c3539326e24d)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ...
00:21:02	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head a860045 · merge a860045	alert, e2e	failed	#6666 (merged)	4	e2e (3) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 00:21:02 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc00100c6c0>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-7hms8hpjlmgg" to finish creating: GET https://rp.j2511104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34872e1a-7b5d-4827-9041-710b4ce9f917 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34872e1a-7b5d-4827-9041-710b4ce9f917", "name": "34872e1a-7b5d-4827-9041-710b4ce9f917", "status": "Failed", "startTime": "2026-08-25T01:11:23.297460639Z", "endTime": "2026-08-25T01:30:30.366070286Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc00100c6c0>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-7hms8hpjlmgg" to finish creating: GET https://rp.j2511104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34872e1a-7b5d-4827-9041-710b4ce9f917 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34872e1a-7b5d-4827-9041-710b4ce9f917", "name": "34872e1a-7b5d-4827-9041-710b4ce9f917", "status": "Failed", "startTime": "2026-08-25T01:11:23.297460639Z", "endTime": "2026-08-25T01:30:30.366070286Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 00:21:02 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260630preview and verify the ingress is internal · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc001048360>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-xjgfszrzk8fd" to finish creating: GET https://rp.j2511104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d5b520fd-de2d-4c04-817a-0cab0589d669 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d5b520fd-de2d-4c04-817a-0cab0589d669", "name": "d5b520fd-de2d-4c04-817a-0cab0589d669", "status": "Failed", "startTime": "2026-08-25T01:10:12.363594861Z", "endTime": "2026-08-25T01:29:20.471660293Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc001048360>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-xjgfszrzk8fd" to finish creating: GET https://rp.j2511104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d5b520fd-de2d-4c04-817a-0cab0589d669 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d5b520fd-de2d-4c04-817a-0cab0589d669", "name": "d5b520fd-de2d-4c04-817a-0cab0589d669", "status": "Failed", "startTime": "2026-08-25T01:10:12.363594861Z", "endTime": "2026-08-25T01:29:20.471660293Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 00:21:02 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-z4pnctblvmpf/cluster-candidate-4-22-bt5tfn should provision Unexpected error: <fmt.wrapError \| 0xc0017de640>: failed to create HCP cluster cluster-candidate-4-22-bt5tfn: failed waiting for cluster="cluster-candidate-4-22-bt5tfn" in resourcegroup="rg-candidate-4-22-z4pnctblvmpf" to finish creating: GET https://rp.j2511104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8a2b3ac-1907-490d-8c60-3586709ab393 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8a2b3ac-1907-490d-8c60-3586709ab393", "name": "c8a2b3ac-1907-490d-8c60-3586709ab393", "status": "Failed", "startTime": "2026-08-25T01:09:52.015224549Z", "endTime": "2026-08-25T01:29:00.623079296Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-z4pnctblvmpf/cluster-candidate-4-22-bt5tfn should provision Unexpected error: <fmt.wrapError \| 0xc0017de640>: failed to create HCP cluster cluster-candidate-4-22-bt5tfn: failed waiting for cluster="cluster-candidate-4-22-bt5tfn" in resourcegroup="rg-candidate-4-22-z4pnctblvmpf" to finish creating: GET https://rp.j2511104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8a2b3ac-1907-490d-8c60-3586709ab393 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8a2b3ac-1907-490d-8c60-3586709ab393", "name": "c8a2b3ac-1907-490d-8c60-3586709ab393", "status": "Failed", "startTime": "2026-08-25T01:09:52.015224549Z", "endTime": "2026-08-25T01:29:00.623079296Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] KubePodNotReady fired 00:21:02 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-08-25T01:22:06Z Ended: 2026-08-25T01:39:09Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2511104-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd9qo43pb87kgb0najuskd4463erq6p-s6z0k3y7v0o2v4a", pod="router-794d4d4f76-jgglj", severity="warning" Description: Pod ocm-arohcpci01-2sd9qo43pb87kgb0najuskd4463erq6p-s6z0k3y7v0o2v4a/router-794d4d4f76-jgglj has been in a non-ready state for longer than 5 minutes.
00:07:21	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 87c9d3c	e2e	failed	★#6662 (open)	1	e2e (1) query-basic: command failed: exit status 1 00:07:21 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.003s. Expected success, but got an error: <errors.errorString \| 0xc0008284f0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[01:06:30.008]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T01:06:27.0754516Z);\nlet _endTime = datetime(2026-08-25T01:06:27.0754522Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xl7j9qn9wjl5'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T01:06:29.9876601Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;ea9f9362-a343-4744-80b3-340fe3cac212\",\r\n \"activityId\": \"59bb71ac-27cb-4cfe-91f1-0dc80dc01c78\",\r\n \"subActivityId\": \"a6ab6f7d-7448-47fd-b473-40d106fd5ddb\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"df117fcb-5da2-4690-9e92-052d46a6d0e1\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;ea9f9362-a343-4744-80b3-340fe3cac212 ARID=59bb71ac-27cb-4cfe-91f1-0dc80dc01c78 \u003e GW.Http.CallContext/59bb71ac-27cb-4cfe-91f1-0dc80dc01c78 \u003e GW.ExecuteQuery/df117fcb-5da2-4690-9e92-052d46a6d0e1 \u003e DN.FE.ExecuteQuery/a6ab6f7d-7448-47fd-b473-40d106fd5ddb)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[01:06:30.057]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T01:06:27.0754516Z);\nlet _endTime = datetime(2026-08-25T01:06:27.0754522Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xl7j9qn9wjl5'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T01:06:29.9876601Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;ea9f9362-a343-4744-80b3-340fe3cac212\",\r\n \"activityId\": \"59bb71ac-27cb-4cfe-91f1-0dc80dc01c78\",\r\n \"subActivityId\": \"a6ab6f7d-7448-47fd-b473-40d106fd5ddb\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"df117fcb-5da2-4690-9e92-052d46a6d0e1\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;ea9f9362-a343-4744-80b3-340fe3cac212 ARID=59bb71ac-27cb-4cfe-91f1-0dc80dc01c78 \u003e GW.Http.CallContext/59bb71ac-27cb-4cfe-91f1-0dc80dc01c78 \u003e GW.ExecuteQuery/df117fcb-5da2-4690-9e92-052d46a6d0e1 \u003e DN.FE.ExecuteQuery/a6ab6f7d-7448-47fd-b473-40d106fd5ddb)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[01:06:33.062]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T01:06:30.1288605Z);\nlet _endTime = datetime(2026-08-25T01:06:30.1288611Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xl7j9qn9wjl5'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T01:06:33.0424541Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;20e6c056-fef5-435a-98cd-ebba63cc75a6\",\r\n \"activityId\": \"d5604ed5-126a-4121-b965-3b7cbf0283f1\",\r\n \"subActivityId\": \"dd36c913-5740-44c2-bd2d-98a620b22e83\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"436ae237-897e-4d8a-81ee-c4b74c7b1d5b\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;20e6c056-fef5-435a-98cd-ebba63cc75a6 ARID=d5604ed5-126a-4121-b965-3b7cbf0283f1 \u003e GW.Http.CallContext/d5604ed5-126a-4121-b965-3b7cbf0283f1 \u003e GW.ExecuteQuery/436ae237-897e-4d8a-81ee-c4b74c7b1d5b \u003e DN.FE.ExecuteQuery/dd36c913-5740-44c2-bd2d-98a620b22e83)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[01:06:33.116]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T01:06:30.1288605Z);\nlet _endTime = datetime(2026-08-25T01:06:30.1288611Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xl7j9qn9wjl5'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T01:06:33.0424541Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;20e6c056-fef5-435a-98cd-ebba63cc75a6\",\r\n \"activityId\": \"d5604ed5-126a-4121-b965-3b7cbf0283f1\",\r\n \"subActivityId\": \"dd36c913-5740-44c2-bd2d-98a620b22e83\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"436ae237-897e-4d8a-81ee-c4b74c7b1d5b\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;20e6c056-fef5-435a-98cd-ebba63cc75a6 ARID=d5604ed5-126a-4121-b965-3b7cbf0283f1 \u003e GW.Http.CallContext/d5604ed5-126a-4121-b965-3b7cbf0283f1 \u003e GW.ExecuteQuery/436ae237-897e-4d8a-81ee-c4b74c7b1d5b \u003e DN.FE.ExecuteQuery/dd36c913-5740-44c2-bd2d-98a620b22e83)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[01:06:39.536]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T01:06:36.3917754Z);\nlet _endTime = datetime(2026-08-25T01:06:36.3917761Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xl7j9qn9wjl5'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T01:06:39.5170179Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;26e617de-c0e4-4d37-b522-a570ccd2733c\",\r\n \"activityId\": \"52f0c2d0-c1d5-4598-b0a6-8ab0bda752b2\",\r\n \"subActivityId\": \"3a59414c-fc4c-4995-b8b4-df986c791422\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"11c42a10-1df6-4d97-8ede-cf662bfe0997\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;26e617de-c0e4-4d37-b522-a570ccd2733c ARID=52f0c2d0-c1d5-4598-b0a6-8ab0bda752b2 \u003e GW.Http.CallContext/52f0c2d0-c1d5-4598-b0a6-8ab0bda752b2 \u003e GW.ExecuteQuery/11c42a10-1df6-4d97-8ede-cf662bfe0997 \u003e DN.FE.ExecuteQuery/3a59414c-fc4c-4995-b8b4-df986c791422)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[01:06:39.639]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T01:06:36.3917754Z);\nlet _endTime = datetime(2026-08-25T01:06:36.3917761Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xl7j9qn9wjl5'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T01:06:39.5170179Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;26e617de-c0e4-4d37-b522-a570ccd2733c\",\r\n \"activityId\": \"52f0c2d0-c1d5-4598-b0a6-8ab0bda752b2\",\r\n \"subActivityId\": \"3a59414c-fc4c-4995-b8b4-df986c791422\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"11c42a10-1df6-4d97-8ede-cf662bfe0997\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;26e617de-c0e4-4d37-b522-a570ccd2733c ARID=52f0c2d0-c1d5-4598-b0a6-8ab0bda752b2 \u003e GW.Http.CallContext/52f0c2d0-c1d5-4598-b0a6-8ab0bda752b2 \u003e GW.ExecuteQuery/11c42a10-1df6-4d97-8ede-cf662bfe0997 \u003e DN.FE.ExecuteQuery/3a59414c-fc4c-4995-b8b4-df986c791422)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:100]: Timed out after 600.003s. Expected success, but got an error: <errors.errorString \| 0xc0008284f0>: must-gather CLI test failures: query-basic: command failed: exit status 1 output: �[37m[01:06:30.008]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T01:06:27.0754516Z);\nlet _endTime = datetime(2026-08-25T01:06:27.0754522Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xl7j9qn9wjl5'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T01:06:29.9876601Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;ea9f9362-a343-4744-80b3-340fe3cac212\",\r\n \"activityId\": \"59bb71ac-27cb-4cfe-91f1-0dc80dc01c78\",\r\n \"subActivityId\": \"a6ab6f7d-7448-47fd-b473-40d106fd5ddb\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"df117fcb-5da2-4690-9e92-052d46a6d0e1\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;ea9f9362-a343-4744-80b3-340fe3cac212 ARID=59bb71ac-27cb-4cfe-91f1-0dc80dc01c78 \u003e GW.Http.CallContext/59bb71ac-27cb-4cfe-91f1-0dc80dc01c78 \u003e GW.ExecuteQuery/df117fcb-5da2-4690-9e92-052d46a6d0e1 \u003e DN.FE.ExecuteQuery/a6ab6f7d-7448-47fd-b473-40d106fd5ddb)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[01:06:30.057]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T01:06:27.0754516Z);\nlet _endTime = datetime(2026-08-25T01:06:27.0754522Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xl7j9qn9wjl5'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T01:06:29.9876601Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;ea9f9362-a343-4744-80b3-340fe3cac212\",\r\n \"activityId\": \"59bb71ac-27cb-4cfe-91f1-0dc80dc01c78\",\r\n \"subActivityId\": \"a6ab6f7d-7448-47fd-b473-40d106fd5ddb\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"df117fcb-5da2-4690-9e92-052d46a6d0e1\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;ea9f9362-a343-4744-80b3-340fe3cac212 ARID=59bb71ac-27cb-4cfe-91f1-0dc80dc01c78 \u003e GW.Http.CallContext/59bb71ac-27cb-4cfe-91f1-0dc80dc01c78 \u003e GW.ExecuteQuery/df117fcb-5da2-4690-9e92-052d46a6d0e1 \u003e DN.FE.ExecuteQuery/a6ab6f7d-7448-47fd-b473-40d106fd5ddb)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-skip-hcp-logs: command failed: exit status 1 output: �[37m[01:06:33.062]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T01:06:30.1288605Z);\nlet _endTime = datetime(2026-08-25T01:06:30.1288611Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xl7j9qn9wjl5'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T01:06:33.0424541Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;20e6c056-fef5-435a-98cd-ebba63cc75a6\",\r\n \"activityId\": \"d5604ed5-126a-4121-b965-3b7cbf0283f1\",\r\n \"subActivityId\": \"dd36c913-5740-44c2-bd2d-98a620b22e83\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"436ae237-897e-4d8a-81ee-c4b74c7b1d5b\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;20e6c056-fef5-435a-98cd-ebba63cc75a6 ARID=d5604ed5-126a-4121-b965-3b7cbf0283f1 \u003e GW.Http.CallContext/d5604ed5-126a-4121-b965-3b7cbf0283f1 \u003e GW.ExecuteQuery/436ae237-897e-4d8a-81ee-c4b74c7b1d5b \u003e DN.FE.ExecuteQuery/dd36c913-5740-44c2-bd2d-98a620b22e83)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[01:06:33.116]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T01:06:30.1288605Z);\nlet _endTime = datetime(2026-08-25T01:06:30.1288611Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xl7j9qn9wjl5'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T01:06:33.0424541Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;20e6c056-fef5-435a-98cd-ebba63cc75a6\",\r\n \"activityId\": \"d5604ed5-126a-4121-b965-3b7cbf0283f1\",\r\n \"subActivityId\": \"dd36c913-5740-44c2-bd2d-98a620b22e83\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"436ae237-897e-4d8a-81ee-c4b74c7b1d5b\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;20e6c056-fef5-435a-98cd-ebba63cc75a6 ARID=d5604ed5-126a-4121-b965-3b7cbf0283f1 \u003e GW.Http.CallContext/d5604ed5-126a-4121-b965-3b7cbf0283f1 \u003e GW.ExecuteQuery/436ae237-897e-4d8a-81ee-c4b74c7b1d5b \u003e DN.FE.ExecuteQuery/dd36c913-5740-44c2-bd2d-98a620b22e83)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m query-collect-systemd-logs: command failed: exit status 1 output: �[37m[01:06:39.536]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{ "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T01:06:36.3917754Z);\nlet _endTime = datetime(2026-08-25T01:06:36.3917761Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xl7j9qn9wjl5'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T01:06:39.5170179Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;26e617de-c0e4-4d37-b522-a570ccd2733c\",\r\n \"activityId\": \"52f0c2d0-c1d5-4598-b0a6-8ab0bda752b2\",\r\n \"subActivityId\": \"3a59414c-fc4c-4995-b8b4-df986c791422\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"11c42a10-1df6-4d97-8ede-cf662bfe0997\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;26e617de-c0e4-4d37-b522-a570ccd2733c ARID=52f0c2d0-c1d5-4598-b0a6-8ab0bda752b2 \u003e GW.Http.CallContext/52f0c2d0-c1d5-4598-b0a6-8ab0bda752b2 \u003e GW.ExecuteQuery/11c42a10-1df6-4d97-8ede-cf662bfe0997 \u003e DN.FE.ExecuteQuery/3a59414c-fc4c-4995-b8b4-df986c791422)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}", "name": "detailedInfraOrchestrationLogs" }�[0m �[37m[01:06:39.639]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{ "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint, With query: let _startTime = datetime(2026-08-24T01:06:36.3917754Z);\nlet _endTime = datetime(2026-08-25T01:06:36.3917761Z);\n// Extract cluster ID from CS logs using cid field\nlet cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where resource_id has 'kusto-logs-xl7j9qn9wjl5'\n \| where isnotempty(log.cid)\n \| distinct tostring(log.cid)\n \| limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n \| where isnotempty(resourceId)\n \| distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"maestro\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id or logStr has_any (maestro_resource_ids)\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"hypershift\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n \| where timestamp between (_startTime .. _endTime)\n \| where namespace_name has \"open-cluster-management-agent\"\n \| extend logStr = tostring(log)\n \| where logStr has cluster_id\n \| project timestamp, container_name, msg = logStr, log\n )\n\| order by timestamp asc(400 Bad Request):\n{\r\n \"error\": {\r\n \"code\": \"General_BadRequest\",\r\n \"message\": \"Request is invalid and cannot be executed.\",\r\n \"@type\": \"Kusto.Data.Exceptions.RelopSemanticException\",\r\n \"@message\": \"Relop semantic error: SEM0026: The arguments array exceeded the allowed limit (allowed=10,000)\",\r\n \"@failureCode\": 400,\r\n \"@context\": {\r\n \"timestamp\": \"2026-08-25T01:06:39.5170179Z\",\r\n \"serviceAlias\": \"HCP-DEV-US-2\",\r\n \"clientRequestId\": \"KGC.execute;26e617de-c0e4-4d37-b522-a570ccd2733c\",\r\n \"activityId\": \"52f0c2d0-c1d5-4598-b0a6-8ab0bda752b2\",\r\n \"subActivityId\": \"3a59414c-fc4c-4995-b8b4-df986c791422\",\r\n \"activityType\": \"DN.FE.ExecuteQuery\",\r\n \"parentActivityId\": \"11c42a10-1df6-4d97-8ede-cf662bfe0997\",\r\n \"activityStack\": \"(Activity stack: CRID=KGC.execute;26e617de-c0e4-4d37-b522-a570ccd2733c ARID=52f0c2d0-c1d5-4598-b0a6-8ab0bda752b2 \u003e GW.Http.CallContext/52f0c2d0-c1d5-4598-b0a6-8ab0bda752b2 \u003e GW.ExecuteQuery/11c42a10-1df6-4d97-8ede-cf662bfe0997 \u003e DN.FE.ExecuteQuery/3a59414c-fc4c-4995-b8b4-df986c791422)\",\r\n \"serviceFarm\": null\r\n },\r\n \"@permanent\": true\r\n }\r\n}" }�[0m ...

Environment: INT

Time (UTC)	Job	Failed at	Result	PR	Failed tests	Details
09:05:43	branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel post-goodmerged PR No additional run metadata captured	unknown	failed	n/a	1	non-artifact-backed failure (no junit artifacts)

Environment: PROD

Time (UTC)	Job	Failed at	Result	PR	Failed tests	Details
15:37:01	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
15:18:13	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	unknown	failed	n/a	1	non-artifact-backed failure (no junit artifacts)
14:56:08	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
14:50:10	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
10:10:43	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	1	e2e (1) ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; provider Microsoft.RedHatOpenShift 10:10:43 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260630preview and verify the ingress is internal · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc00093afe0>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-bm5z6ztxsxcf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/56042e31-0efc-4287-925a-c64efa21388d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/56042e31-0efc-4287-925a-c64efa21388d", "name": "56042e31-0efc-4287-925a-c64efa21388d", "status": "Failed", "startTime": "2026-08-25T10:30:08.889821647Z", "endTime": "2026-08-25T10:49:13.717723647Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc00093afe0>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-bm5z6ztxsxcf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/56042e31-0efc-4287-925a-c64efa21388d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/56042e31-0efc-4287-925a-c64efa21388d", "name": "56042e31-0efc-4287-925a-c64efa21388d", "status": "Failed", "startTime": "2026-08-25T10:30:08.889821647Z", "endTime": "2026-08-25T10:49:13.717723647Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted" } } -------------------------------------------------------------------------------- ... occurred
07:06:40	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	2	e2e (2) ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; provider Microsoft.RedHatOpenShift 07:06:40 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc0006d1700>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-dnqrhz4h5n9n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/26ec1bc7-0ddd-42ef-916c-12f4d896455c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/26ec1bc7-0ddd-42ef-916c-12f4d896455c", "name": "26ec1bc7-0ddd-42ef-916c-12f4d896455c", "status": "Failed", "startTime": "2026-08-25T07:34:58.109914662Z", "endTime": "2026-08-25T07:53:59.934466955Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc0006d1700>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-dnqrhz4h5n9n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/26ec1bc7-0ddd-42ef-916c-12f4d896455c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/26ec1bc7-0ddd-42ef-916c-12f4d896455c", "name": "26ec1bc7-0ddd-42ef-916c-12f4d896455c", "status": "Failed", "startTime": "2026-08-25T07:34:58.109914662Z", "endTime": "2026-08-25T07:53:59.934466955Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; provider Microsoft.RedHatOpenShift 07:06:40 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc000a308e0>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-ks6pxxnlhsjm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fc3b6654-559a-489d-a192-908217a06b8f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fc3b6654-559a-489d-a192-908217a06b8f", "name": "fc3b6654-559a-489d-a192-908217a06b8f", "status": "Failed", "startTime": "2026-08-25T07:33:44.141359446Z", "endTime": "2026-08-25T07:52:49.875150436Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc000a308e0>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-ks6pxxnlhsjm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fc3b6654-559a-489d-a192-908217a06b8f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fc3b6654-559a-489d-a192-908217a06b8f", "name": "fc3b6654-559a-489d-a192-908217a06b8f", "status": "Failed", "startTime": "2026-08-25T07:33:44.141359446Z", "endTime": "2026-08-25T07:52:49.875150436Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted" } } -------------------------------------------------------------------------------- ... occurred
02:00:55	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	5	e2e (5) timeout during CreateHCPClusterAndWait; context deadline exceeded 02:00:55 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-zmwf7l76v4hv/cluster-candidate-5-0-p2dtvg should provision Unexpected error: <fmt.wrapErrors \| 0xc000a42420>: failed to create HCP cluster cluster-candidate-5-0-p2dtvg, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-p2dtvg in resource group rg-candidate-5-0-zmwf7l76v4hv, error: failed waiting for cluster="cluster-candidate-5-0-p2dtvg" in resourcegroup="rg-candidate-5-0-zmwf7l76v4hv" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-p2dtvg in resource group rg-candidate-5-0-zmwf7l76v4hv, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-zmwf7l76v4hv/cluster-candidate-5-0-p2dtvg should provision Unexpected error: <fmt.wrapErrors \| 0xc000a42420>: failed to create HCP cluster cluster-candidate-5-0-p2dtvg, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-p2dtvg in resource group rg-candidate-5-0-zmwf7l76v4hv, error: failed waiting for cluster="cluster-candidate-5-0-p2dtvg" in resourcegroup="rg-candidate-5-0-zmwf7l76v4hv" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-p2dtvg in resource group rg-candidate-5-0-zmwf7l76v4hv, error: context deadline exceeded ... occurred ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.21": must be at least 4.21.30; provider Microsoft.RedHatOpenShift 02:00:55 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-21-hc729b" to minor version 4.21 Unexpected error: <exported.ResponseError \| 0xc000cc7e30>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-4j8dsngwmfpg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-hc729b -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": may not decrease from 4.21.29", "target": "properties.version.id" }, { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be at least 4.21.30", "target": "properties.version.id" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-21-hc729b" to minor version 4.21 Unexpected error: <exported.ResponseError \| 0xc000cc7e30>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-4j8dsngwmfpg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-hc729b -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": may not decrease from 4.21.29", "target": "properties.version.id" }, { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be at least 4.21.30", "target": "properties.version.id" } ] } } -------------------------------------------------------------------------------- ... occurred VerifyHostedControlPlaneZStreamUpgradeOnly(initial=5 02:00:55 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:153]: Timed out after 2700.008s. Expected success, but got an error: <errors.joinError \| 0xc000b1c2d0>: VerifyHostedControlPlaneZStreamUpgradeOnly(initial=5.0.0-ec.5) failed: install version "5.0.0-ec.5" not found in clusterversion/version status.history; cannot confirm the z-stream upgrade started from it ... fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:153]: Timed out after 2700.008s. Expected success, but got an error: <errors.joinError \| 0xc000b1c2d0>: VerifyHostedControlPlaneZStreamUpgradeOnly(initial=5.0.0-ec.5) failed: install version "5.0.0-ec.5" not found in clusterversion/version status.history; cannot confirm the z-stream upgrade started from it ... ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.20": must be at least 4.20.35; provider Microsoft.RedHatOpenShift 02:00:55 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-20-2m6hjk" to minor version 4.20 Unexpected error: <exported.ResponseError \| 0xc00144a750>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-6dbxgwtzxfsh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-2m6hjk -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20\": may not decrease from 4.20.34", "target": "properties.version.id" }, { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20\": must be at least 4.20.35", "target": "properties.version.id" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-20-2m6hjk" to minor version 4.20 Unexpected error: <exported.ResponseError \| 0xc00144a750>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-6dbxgwtzxfsh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-2m6hjk -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20\": may not decrease from 4.20.34", "target": "properties.version.id" }, { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20\": must be at least 4.20.35", "target": "properties.version.id" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.22": must be at least 4.22.10; provider Microsoft.RedHatOpenShift 02:00:55 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-22-qp64bq" to minor version 4.22 Unexpected error: <exported.ResponseError \| 0xc000a2bec0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-dqpzmwzqshp6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-qp64bq -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Invalid value: \"4.22\": may not decrease from 4.22.10", "target": "properties.version.id" }, { "code": "InvalidRequestContent", "message": "Invalid value: \"4.22\": must be at least 4.22.10", "target": "properties.version.id" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-22-qp64bq" to minor version 4.22 Unexpected error: <exported.ResponseError \| 0xc000a2bec0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-dqpzmwzqshp6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-qp64bq -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Invalid value: \"4.22\": may not decrease from 4.22.10", "target": "properties.version.id" }, { "code": "InvalidRequestContent", "message": "Invalid value: \"4.22\": must be at least 4.22.10", "target": "properties.version.id" } ] } } -------------------------------------------------------------------------------- ... occurred
01:17:13	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a

Environment: STG

Time (UTC)	Job	Failed at	Result	PR	Failed tests	Details
04:35:26	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
02:00:55	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel post-goodmerged PR No additional run metadata captured	unknown	failed	n/a	1	non-artifact-backed failure (no junit artifacts)
00:59:50	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a