CIHealth Run Log

Environment: PROD

Time (UTC)	Job	Failed at	Result	PR	Failed tests	Details
23:00:07	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	4	e2e (4) timeout during CreateHCPClusterAndWait; context deadline exceeded 23:00:07 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapErrors \| 0xc002780840>: failed to create HCP cluster private-kas, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-272j4k46kvrl, error: failed waiting for cluster="private-kas" in resourcegroup="private-kas-272j4k46kvrl" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-272j4k46kvrl, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapErrors \| 0xc002780840>: failed to create HCP cluster private-kas, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-272j4k46kvrl, error: failed waiting for cluster="private-kas" in resourcegroup="private-kas-272j4k46kvrl" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-272j4k46kvrl, error: context deadline exceeded ... occurred ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'uksouth' and API version '2026-06-30-preview' for type 'locations/hcpOperationStatuses'. The supported api-versions are '2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'centraluseuap, eastus2euap, uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2'.; provider Microsoft.RedHatOpenShift 23:00:07 UTC · failed at e2e · test FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:120]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc000a30180>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-nkkjqglbtkz6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f40d5702-8f9c-44c7-adc1-5d00590f3971 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: NoRegisteredProviderFound -------------------------------------------------------------------------------- { "error": { "code": "NoRegisteredProviderFound", "message": "No registered resource provider found for location 'uksouth' and API version '2026-06-30-preview' for type 'locations/hcpOperationStatuses'. The supported api-versions are '2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'centraluseuap, eastus2euap, uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2'." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:120]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc000a30180>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-nkkjqglbtkz6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f40d5702-8f9c-44c7-adc1-5d00590f3971 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: NoRegisteredProviderFound -------------------------------------------------------------------------------- { "error": { "code": "NoRegisteredProviderFound", "message": "No registered resource provider found for location 'uksouth' and API version '2026-06-30-preview' for type 'locations/hcpOperationStatuses'. The supported api-versions are '2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'centraluseuap, eastus2euap, uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2'." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 23:00:07 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1 Unexpected error: <fmt.wrapError \| 0xc0009ad7e0>: failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-z7fl6rs7cjg5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b557842c-42a5-4d65-80d1-401050420a8d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b557842c-42a5-4d65-80d1-401050420a8d", "name": "b557842c-42a5-4d65-80d1-401050420a8d", "status": "Failed", "startTime": "2026-08-06T23:42:49.092257041Z", "endTime": "2026-08-07T00:02:00.144502022Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1 Unexpected error: <fmt.wrapError \| 0xc0009ad7e0>: failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-z7fl6rs7cjg5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b557842c-42a5-4d65-80d1-401050420a8d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b557842c-42a5-4d65-80d1-401050420a8d", "name": "b557842c-42a5-4d65-80d1-401050420a8d", "status": "Failed", "startTime": "2026-08-06T23:42:49.092257041Z", "endTime": "2026-08-07T00:02:00.144502022Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 23:00:07 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc00171c228>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-wvptdr" in resourcegroup="rg-zstream-upgrade-4-21-bvlh6ndvpmbz" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8813fe71-a996-402a-8b22-5a2a07f190dc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8813fe71-a996-402a-8b22-5a2a07f190dc", "name": "8813fe71-a996-402a-8b22-5a2a07f190dc", "status": "Failed", "startTime": "2026-08-06T23:41:52.408330345Z", "endTime": "2026-08-07T00:06:00.233442021Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s1cmcbdh4et6l3hlfi6mpmgpaik517c still exists (deletion dispatched at 2026-08-06T23:41:53Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc00171c228>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-wvptdr" in resourcegroup="rg-zstream-upgrade-4-21-bvlh6ndvpmbz" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8813fe71-a996-402a-8b22-5a2a07f190dc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8813fe71-a996-402a-8b22-5a2a07f190dc", "name": "8813fe71-a996-402a-8b22-5a2a07f190dc", "status": "Failed", "startTime": "2026-08-06T23:41:52.408330345Z", "endTime": "2026-08-07T00:06:00.233442021Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s1cmcbdh4et6l3hlfi6mpmgpaik517c still exists (deletion dispatched at 2026-08-06T23:41:53Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred
21:37:13	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
18:58:46	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
18:58:04	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
18:54:30	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	1	e2e (1) ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 18:54:30 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:328]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster" Unexpected error: <errors.joinError \| 0xc000e90288>: VerifyNodesReady failed: no nodes found in the cluster VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-2tgsqq2jndsw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/60668361-99db-4918-b1e5-1dd336777928 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/60668361-99db-4918-b1e5-1dd336777928", "name": "60668361-99db-4918-b1e5-1dd336777928", "status": "Failed", "startTime": "2026-08-06T19:32:55.875583035Z", "endTime": "2026-08-06T19:51:56.785806181Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:328]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster" Unexpected error: <errors.joinError \| 0xc000e90288>: VerifyNodesReady failed: no nodes found in the cluster VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-2tgsqq2jndsw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/60668361-99db-4918-b1e5-1dd336777928 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/60668361-99db-4918-b1e5-1dd336777928", "name": "60668361-99db-4918-b1e5-1dd336777928", "status": "Failed", "startTime": "2026-08-06T19:32:55.875583035Z", "endTime": "2026-08-06T19:51:56.785806181Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred
14:08:38	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
14:04:40	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
14:02:57	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
13:56:02	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
12:27:57	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	29	e2e (29) ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:27:57 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:217]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0xc000e21180>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-wm5j8g7bxx2m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/47987963-3e83-4032-867d-1e91993059bf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/47987963-3e83-4032-867d-1e91993059bf", "name": "47987963-3e83-4032-867d-1e91993059bf", "status": "Failed", "startTime": "2026-08-06T14:06:15.739222219Z", "endTime": "2026-08-06T14:25:32.992085923Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:217]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0xc000e21180>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-wm5j8g7bxx2m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/47987963-3e83-4032-867d-1e91993059bf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/47987963-3e83-4032-867d-1e91993059bf", "name": "47987963-3e83-4032-867d-1e91993059bf", "status": "Failed", "startTime": "2026-08-06T14:06:15.739222219Z", "endTime": "2026-08-06T14:25:32.992085923Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:27:57 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:462]: failed to create HCP cluster np-skip-minor-h8m8gn at version 4.22.8 Unexpected error: <fmt.wrapError \| 0xc0011ba160>: failed to create HCP cluster np-skip-minor-h8m8gn: failed waiting for cluster="np-skip-minor-h8m8gn" in resourcegroup="rg-np-skip-minor-h8m8gn-mmthppt2pm7w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/045886da-6fff-4fbb-938f-87cdb4e85bad -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/045886da-6fff-4fbb-938f-87cdb4e85bad", "name": "045886da-6fff-4fbb-938f-87cdb4e85bad", "status": "Failed", "startTime": "2026-08-06T14:02:42.413063766Z", "endTime": "2026-08-06T14:21:42.765300575Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:462]: failed to create HCP cluster np-skip-minor-h8m8gn at version 4.22.8 Unexpected error: <fmt.wrapError \| 0xc0011ba160>: failed to create HCP cluster np-skip-minor-h8m8gn: failed waiting for cluster="np-skip-minor-h8m8gn" in resourcegroup="rg-np-skip-minor-h8m8gn-mmthppt2pm7w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/045886da-6fff-4fbb-938f-87cdb4e85bad -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/045886da-6fff-4fbb-938f-87cdb4e85bad", "name": "045886da-6fff-4fbb-938f-87cdb4e85bad", "status": "Failed", "startTime": "2026-08-06T14:02:42.413063766Z", "endTime": "2026-08-06T14:21:42.765300575Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:27:57 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0xc000de74e0>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-ts8pl2pswsxq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/17591ace-40e3-4096-ab61-8932d25ea783 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/17591ace-40e3-4096-ab61-8932d25ea783", "name": "17591ace-40e3-4096-ab61-8932d25ea783", "status": "Failed", "startTime": "2026-08-06T13:27:45.606306967Z", "endTime": "2026-08-06T13:46:50.497502059Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0xc000de74e0>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-ts8pl2pswsxq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/17591ace-40e3-4096-ab61-8932d25ea783 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/17591ace-40e3-4096-ab61-8932d25ea783", "name": "17591ace-40e3-4096-ab61-8932d25ea783", "status": "Failed", "startTime": "2026-08-06T13:27:45.606306967Z", "endTime": "2026-08-06T13:46:50.497502059Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:27:57 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:741]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000686c20>: failed to create HCP cluster np-dg-minor-rlg5t7: failed waiting for cluster="np-dg-minor-rlg5t7" in resourcegroup="rg-np-dg-minor-rlg5t7-rrc9jm2hdf6d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fdf3be45-0611-45f9-93a3-33c90642bf28 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fdf3be45-0611-45f9-93a3-33c90642bf28", "name": "fdf3be45-0611-45f9-93a3-33c90642bf28", "status": "Failed", "startTime": "2026-08-06T14:03:30.092901342Z", "endTime": "2026-08-06T14:22:32.789219091Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:741]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000686c20>: failed to create HCP cluster np-dg-minor-rlg5t7: failed waiting for cluster="np-dg-minor-rlg5t7" in resourcegroup="rg-np-dg-minor-rlg5t7-rrc9jm2hdf6d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fdf3be45-0611-45f9-93a3-33c90642bf28 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fdf3be45-0611-45f9-93a3-33c90642bf28", "name": "fdf3be45-0611-45f9-93a3-33c90642bf28", "status": "Failed", "startTime": "2026-08-06T14:03:30.092901342Z", "endTime": "2026-08-06T14:22:32.789219091Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 12:27:57 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-22-tg2v4gkq6vnr/cluster-candidate-4-22-p7kqt2 should provision Unexpected error: <fmt.wrapError \| 0xc000f10c40>: failed to create HCP cluster cluster-candidate-4-22-p7kqt2: failed waiting for cluster="cluster-candidate-4-22-p7kqt2" in resourcegroup="rg-candidate-4-22-tg2v4gkq6vnr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/eb16ace1-2bdd-451a-9d78-7250d76ae24b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/eb16ace1-2bdd-451a-9d78-7250d76ae24b", "name": "eb16ace1-2bdd-451a-9d78-7250d76ae24b", "status": "Failed", "startTime": "2026-08-06T13:14:36.88824918Z", "endTime": "2026-08-06T13:33:40.344451949Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-22-tg2v4gkq6vnr/cluster-candidate-4-22-p7kqt2 should provision Unexpected error: <fmt.wrapError \| 0xc000f10c40>: failed to create HCP cluster cluster-candidate-4-22-p7kqt2: failed waiting for cluster="cluster-candidate-4-22-p7kqt2" in resourcegroup="rg-candidate-4-22-tg2v4gkq6vnr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/eb16ace1-2bdd-451a-9d78-7250d76ae24b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/eb16ace1-2bdd-451a-9d78-7250d76ae24b", "name": "eb16ace1-2bdd-451a-9d78-7250d76ae24b", "status": "Failed", "startTime": "2026-08-06T13:14:36.88824918Z", "endTime": "2026-08-06T13:33:40.344451949Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:27:57 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-v22lmz with version 4.20.32 Unexpected error: <fmt.wrapError \| 0xc000880740>: failed to create HCP cluster np-version-upgrade-cluster-v22lmz: failed waiting for cluster="np-version-upgrade-cluster-v22lmz" in resourcegroup="rg-np-version-upgrade-v22lmz-dv89nwqtszzk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b382466b-61e3-429b-a11e-9e3842470fc6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b382466b-61e3-429b-a11e-9e3842470fc6", "name": "b382466b-61e3-429b-a11e-9e3842470fc6", "status": "Failed", "startTime": "2026-08-06T13:50:27.651550221Z", "endTime": "2026-08-06T14:09:31.872279557Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-v22lmz with version 4.20.32 Unexpected error: <fmt.wrapError \| 0xc000880740>: failed to create HCP cluster np-version-upgrade-cluster-v22lmz: failed waiting for cluster="np-version-upgrade-cluster-v22lmz" in resourcegroup="rg-np-version-upgrade-v22lmz-dv89nwqtszzk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b382466b-61e3-429b-a11e-9e3842470fc6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b382466b-61e3-429b-a11e-9e3842470fc6", "name": "b382466b-61e3-429b-a11e-9e3842470fc6", "status": "Failed", "startTime": "2026-08-06T13:50:27.651550221Z", "endTime": "2026-08-06T14:09:31.872279557Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:27:57 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc000d9b6a0>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-ccwxckwntr6l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/68677096-9fc1-47ab-aa3f-959f9dd830ad -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/68677096-9fc1-47ab-aa3f-959f9dd830ad", "name": "68677096-9fc1-47ab-aa3f-959f9dd830ad", "status": "Failed", "startTime": "2026-08-06T13:24:48.671191592Z", "endTime": "2026-08-06T13:43:50.420986579Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc000d9b6a0>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-ccwxckwntr6l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/68677096-9fc1-47ab-aa3f-959f9dd830ad -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/68677096-9fc1-47ab-aa3f-959f9dd830ad", "name": "68677096-9fc1-47ab-aa3f-959f9dd830ad", "status": "Failed", "startTime": "2026-08-06T13:24:48.671191592Z", "endTime": "2026-08-06T13:43:50.420986579Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:27:57 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:314]: failed to create HCP cluster np-noedge-l2gkzv Unexpected error: <fmt.wrapError \| 0xc00078b540>: failed to create HCP cluster np-noedge-l2gkzv: failed waiting for cluster="np-noedge-l2gkzv" in resourcegroup="rg-np-noedge-l2gkzv-5wbj828vxrrx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/01bbf857-2cca-4bc8-8a4e-9a9270d4dc75 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/01bbf857-2cca-4bc8-8a4e-9a9270d4dc75", "name": "01bbf857-2cca-4bc8-8a4e-9a9270d4dc75", "status": "Failed", "startTime": "2026-08-06T14:01:01.527104846Z", "endTime": "2026-08-06T14:20:12.805550311Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:314]: failed to create HCP cluster np-noedge-l2gkzv Unexpected error: <fmt.wrapError \| 0xc00078b540>: failed to create HCP cluster np-noedge-l2gkzv: failed waiting for cluster="np-noedge-l2gkzv" in resourcegroup="rg-np-noedge-l2gkzv-5wbj828vxrrx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/01bbf857-2cca-4bc8-8a4e-9a9270d4dc75 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/01bbf857-2cca-4bc8-8a4e-9a9270d4dc75", "name": "01bbf857-2cca-4bc8-8a4e-9a9270d4dc75", "status": "Failed", "startTime": "2026-08-06T14:01:01.527104846Z", "endTime": "2026-08-06T14:20:12.805550311Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 12:27:57 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:120]: failed to get admin REST config for cluster "cluster-zstream-4-19-crm6vw" Unexpected error: <fmt.wrapErrors \| 0xc000939710>: failed waiting for hcpCluster="cluster-zstream-4-19-crm6vw" in resourcegroup="rg-zstream-upgrade-4-19-2jfbcdvqrzbm" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-crm6vw in resource group rg-zstream-upgrade-4-19-2jfbcdvqrzbm, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:120]: failed to get admin REST config for cluster "cluster-zstream-4-19-crm6vw" Unexpected error: <fmt.wrapErrors \| 0xc000939710>: failed waiting for hcpCluster="cluster-zstream-4-19-crm6vw" in resourcegroup="rg-zstream-upgrade-4-19-2jfbcdvqrzbm" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-crm6vw in resource group rg-zstream-upgrade-4-19-2jfbcdvqrzbm, error: context deadline exceeded ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:27:57 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0xc001116360>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-jplb8l9bzh7n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7303a152-653d-4091-80cf-f62f456fb7b9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7303a152-653d-4091-80cf-f62f456fb7b9", "name": "7303a152-653d-4091-80cf-f62f456fb7b9", "status": "Failed", "startTime": "2026-08-06T13:25:47.63498122Z", "endTime": "2026-08-06T13:44:50.425486913Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0xc001116360>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-jplb8l9bzh7n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7303a152-653d-4091-80cf-f62f456fb7b9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7303a152-653d-4091-80cf-f62f456fb7b9", "name": "7303a152-653d-4091-80cf-f62f456fb7b9", "status": "Failed", "startTime": "2026-08-06T13:25:47.63498122Z", "endTime": "2026-08-06T13:44:50.425486913Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:27:57 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0xc000849d40>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-f2kqg69vsqnf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/38be9e5f-4202-47f2-ba3d-563b8721207a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/38be9e5f-4202-47f2-ba3d-563b8721207a", "name": "38be9e5f-4202-47f2-ba3d-563b8721207a", "status": "Failed", "startTime": "2026-08-06T13:23:25.670668106Z", "endTime": "2026-08-06T13:42:30.41053575Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0xc000849d40>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-f2kqg69vsqnf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/38be9e5f-4202-47f2-ba3d-563b8721207a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/38be9e5f-4202-47f2-ba3d-563b8721207a", "name": "38be9e5f-4202-47f2-ba3d-563b8721207a", "status": "Failed", "startTime": "2026-08-06T13:23:25.670668106Z", "endTime": "2026-08-06T13:42:30.41053575Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 12:27:57 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:174]: failed to create node pool "np-zjfs89" for cluster "cluster-candidate-4-20-zjfs89" Unexpected error: <fmt.wrapError \| 0xc000d491a0>: failed to create NodePool np-zjfs89: failed waiting for nodepool="np-zjfs89" for cluster "cluster-candidate-4-20-zjfs89" in resourcegroup="rg-candidate-4-20-lwxfdj6b82tz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b242787d-f97b-4793-82e7-f4a360d7884c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b242787d-f97b-4793-82e7-f4a360d7884c", "name": "b242787d-f97b-4793-82e7-f4a360d7884c", "status": "Failed", "startTime": "2026-08-06T13:22:17.428651995Z", "endTime": "2026-08-06T13:41:30.639404623Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:174]: failed to create node pool "np-zjfs89" for cluster "cluster-candidate-4-20-zjfs89" Unexpected error: <fmt.wrapError \| 0xc000d491a0>: failed to create NodePool np-zjfs89: failed waiting for nodepool="np-zjfs89" for cluster "cluster-candidate-4-20-zjfs89" in resourcegroup="rg-candidate-4-20-lwxfdj6b82tz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b242787d-f97b-4793-82e7-f4a360d7884c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b242787d-f97b-4793-82e7-f4a360d7884c", "name": "b242787d-f97b-4793-82e7-f4a360d7884c", "status": "Failed", "startTime": "2026-08-06T13:22:17.428651995Z", "endTime": "2026-08-06T13:41:30.639404623Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: externalAuths, serviceProviderClusters; provider Microsoft.RedHatOpenShift 12:27:57 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc001d84300>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-7zlftg2lbn8z" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/662e7bf9-43c4-4c69-b7d4-c84c5e38b0ea -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/662e7bf9-43c4-4c69-b7d4-c84c5e38b0ea", "name": "662e7bf9-43c4-4c69-b7d4-c84c5e38b0ea", "status": "Failed", "startTime": "2026-08-06T13:21:44.42927409Z", "endTime": "2026-08-06T13:45:50.758225457Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s139v1qlkvi8boiqf5sq004bbeikr8v still exists (deletion dispatched at 2026-08-06T13:21:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc001d84300>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-7zlftg2lbn8z" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/662e7bf9-43c4-4c69-b7d4-c84c5e38b0ea -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/662e7bf9-43c4-4c69-b7d4-c84c5e38b0ea", "name": "662e7bf9-43c4-4c69-b7d4-c84c5e38b0ea", "status": "Failed", "startTime": "2026-08-06T13:21:44.42927409Z", "endTime": "2026-08-06T13:45:50.758225457Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s139v1qlkvi8boiqf5sq004bbeikr8v still exists (deletion dispatched at 2026-08-06T13:21:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:27:57 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster Unexpected error: <fmt.wrapError \| 0xc000a9c6a0>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-jkhl9hhgd9nc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8eff199a-e62d-435d-aee5-867005cccb86 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8eff199a-e62d-435d-aee5-867005cccb86", "name": "8eff199a-e62d-435d-aee5-867005cccb86", "status": "Failed", "startTime": "2026-08-06T13:26:19.442548042Z", "endTime": "2026-08-06T13:45:30.447966088Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster Unexpected error: <fmt.wrapError \| 0xc000a9c6a0>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-jkhl9hhgd9nc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8eff199a-e62d-435d-aee5-867005cccb86 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8eff199a-e62d-435d-aee5-867005cccb86", "name": "8eff199a-e62d-435d-aee5-867005cccb86", "status": "Failed", "startTime": "2026-08-06T13:26:19.442548042Z", "endTime": "2026-08-06T13:45:30.447966088Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift 12:27:57 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-20-zfn6l2" with version 4.20.28 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000d51220>: failed to create HCP cluster cluster-zstream-4-20-zfn6l2: failed waiting for cluster="cluster-zstream-4-20-zfn6l2" in resourcegroup="rg-zstream-upgrade-4-20-2pwfwd7cj72b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/88e04dc1-f997-4ecd-9971-75dc18ced266 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/88e04dc1-f997-4ecd-9971-75dc18ced266", "name": "88e04dc1-f997-4ecd-9971-75dc18ced266", "status": "Failed", "startTime": "2026-08-06T13:18:03.306664145Z", "endTime": "2026-08-06T13:37:10.59559601Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-20-zfn6l2" with version 4.20.28 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000d51220>: failed to create HCP cluster cluster-zstream-4-20-zfn6l2: failed waiting for cluster="cluster-zstream-4-20-zfn6l2" in resourcegroup="rg-zstream-upgrade-4-20-2pwfwd7cj72b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/88e04dc1-f997-4ecd-9971-75dc18ced266 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/88e04dc1-f997-4ecd-9971-75dc18ced266", "name": "88e04dc1-f997-4ecd-9971-75dc18ced266", "status": "Failed", "startTime": "2026-08-06T13:18:03.306664145Z", "endTime": "2026-08-06T13:37:10.59559601Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:27:57 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-9dwj48 with version 4.21.27 Unexpected error: <fmt.wrapError \| 0xc000fadf20>: failed to create HCP cluster np-version-upgrade-cluster-9dwj48: failed waiting for cluster="np-version-upgrade-cluster-9dwj48" in resourcegroup="rg-np-version-upgrade-9dwj48-gvwgsnfm4p5r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/77d206f8-ee5c-4108-b10e-e10bed6f02a1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/77d206f8-ee5c-4108-b10e-e10bed6f02a1", "name": "77d206f8-ee5c-4108-b10e-e10bed6f02a1", "status": "Failed", "startTime": "2026-08-06T13:45:47.06733678Z", "endTime": "2026-08-06T14:04:51.795324203Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-9dwj48 with version 4.21.27 Unexpected error: <fmt.wrapError \| 0xc000fadf20>: failed to create HCP cluster np-version-upgrade-cluster-9dwj48: failed waiting for cluster="np-version-upgrade-cluster-9dwj48" in resourcegroup="rg-np-version-upgrade-9dwj48-gvwgsnfm4p5r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/77d206f8-ee5c-4108-b10e-e10bed6f02a1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/77d206f8-ee5c-4108-b10e-e10bed6f02a1", "name": "77d206f8-ee5c-4108-b10e-e10bed6f02a1", "status": "Failed", "startTime": "2026-08-06T13:45:47.06733678Z", "endTime": "2026-08-06T14:04:51.795324203Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:27:57 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:270]: failed to create HCP cluster shoebox-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000a1c7e0>: failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-p6k5qdqwc88z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b09a508d-d159-4b59-bd6c-31a88a545881 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b09a508d-d159-4b59-bd6c-31a88a545881", "name": "b09a508d-d159-4b59-bd6c-31a88a545881", "status": "Failed", "startTime": "2026-08-06T14:05:44.41212661Z", "endTime": "2026-08-06T14:24:52.800922454Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:270]: failed to create HCP cluster shoebox-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000a1c7e0>: failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-p6k5qdqwc88z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b09a508d-d159-4b59-bd6c-31a88a545881 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b09a508d-d159-4b59-bd6c-31a88a545881", "name": "b09a508d-d159-4b59-bd6c-31a88a545881", "status": "Failed", "startTime": "2026-08-06T14:05:44.41212661Z", "endTime": "2026-08-06T14:24:52.800922454Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded 12:27:57 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc000b2aa68>: failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-cluster-back-version-tkgzckfhvn9s", caused by: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-tkgzckfhvn9s" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-tkgzckfhvn9s, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-tkgzckfhvn9s" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-tkgzckfhvn9s, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc000b2aa68>: failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-cluster-back-version-tkgzckfhvn9s", caused by: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-tkgzckfhvn9s" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-tkgzckfhvn9s, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-tkgzckfhvn9s" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-tkgzckfhvn9s, error: context deadline exceeded ... occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 12:27:57 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:131]: failed to get admin REST config for cluster "cluster-candidate-4-21-4tlk2m" Unexpected error: <fmt.wrapErrors \| 0xc0008ebb00>: failed waiting for hcpCluster="cluster-candidate-4-21-4tlk2m" in resourcegroup="rg-candidate-4-21-gwbm5ljx4jpp" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-candidate-4-21-4tlk2m in resource group rg-candidate-4-21-gwbm5ljx4jpp, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:131]: failed to get admin REST config for cluster "cluster-candidate-4-21-4tlk2m" Unexpected error: <fmt.wrapErrors \| 0xc0008ebb00>: failed waiting for hcpCluster="cluster-candidate-4-21-4tlk2m" in resourcegroup="rg-candidate-4-21-gwbm5ljx4jpp" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-candidate-4-21-4tlk2m in resource group rg-candidate-4-21-gwbm5ljx4jpp, error: context deadline exceeded ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift 12:27:57 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-22-qgwpct" with version 4.22.6 on candidate channel Unexpected error: <fmt.wrapError \| 0xc0001a9580>: failed to create HCP cluster cluster-zstream-4-22-qgwpct: failed waiting for cluster="cluster-zstream-4-22-qgwpct" in resourcegroup="rg-zstream-upgrade-4-22-sq8l54wqsbkm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4bbc9d45-d885-4a5e-a947-9f6cc88c4261 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4bbc9d45-d885-4a5e-a947-9f6cc88c4261", "name": "4bbc9d45-d885-4a5e-a947-9f6cc88c4261", "status": "Failed", "startTime": "2026-08-06T13:19:49.664906309Z", "endTime": "2026-08-06T13:38:50.306742743Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-22-qgwpct" with version 4.22.6 on candidate channel Unexpected error: <fmt.wrapError \| 0xc0001a9580>: failed to create HCP cluster cluster-zstream-4-22-qgwpct: failed waiting for cluster="cluster-zstream-4-22-qgwpct" in resourcegroup="rg-zstream-upgrade-4-22-sq8l54wqsbkm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4bbc9d45-d885-4a5e-a947-9f6cc88c4261 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4bbc9d45-d885-4a5e-a947-9f6cc88c4261", "name": "4bbc9d45-d885-4a5e-a947-9f6cc88c4261", "status": "Failed", "startTime": "2026-08-06T13:19:49.664906309Z", "endTime": "2026-08-06T13:38:50.306742743Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift 12:27:57 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:182]: failed to create HCP cluster "cp-ystream-upgrade-4-21-zpdmdp" with install version 4.20.32 Unexpected error: <fmt.wrapError \| 0xc00083a720>: failed to create HCP cluster cp-ystream-upgrade-4-21-zpdmdp: failed waiting for cluster="cp-ystream-upgrade-4-21-zpdmdp" in resourcegroup="rg-cp-ystream-upgrade-4-21-nt59ffl77spl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5bb285a7-7ce1-4027-9552-2a08322f2e58 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5bb285a7-7ce1-4027-9552-2a08322f2e58", "name": "5bb285a7-7ce1-4027-9552-2a08322f2e58", "status": "Failed", "startTime": "2026-08-06T13:20:54.106332052Z", "endTime": "2026-08-06T13:40:00.506003922Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:182]: failed to create HCP cluster "cp-ystream-upgrade-4-21-zpdmdp" with install version 4.20.32 Unexpected error: <fmt.wrapError \| 0xc00083a720>: failed to create HCP cluster cp-ystream-upgrade-4-21-zpdmdp: failed waiting for cluster="cp-ystream-upgrade-4-21-zpdmdp" in resourcegroup="rg-cp-ystream-upgrade-4-21-nt59ffl77spl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5bb285a7-7ce1-4027-9552-2a08322f2e58 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5bb285a7-7ce1-4027-9552-2a08322f2e58", "name": "5bb285a7-7ce1-4027-9552-2a08322f2e58", "status": "Failed", "startTime": "2026-08-06T13:20:54.106332052Z", "endTime": "2026-08-06T13:40:00.506003922Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:27:57 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster Unexpected error: <fmt.wrapError \| 0xc00165cba0>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-b2xd2cfggh75" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1b4f6d3-9523-41a1-b21f-c572ab6386df -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1b4f6d3-9523-41a1-b21f-c572ab6386df", "name": "d1b4f6d3-9523-41a1-b21f-c572ab6386df", "status": "Failed", "startTime": "2026-08-06T14:06:00.450328897Z", "endTime": "2026-08-06T14:25:02.805147479Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster Unexpected error: <fmt.wrapError \| 0xc00165cba0>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-b2xd2cfggh75" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1b4f6d3-9523-41a1-b21f-c572ab6386df -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1b4f6d3-9523-41a1-b21f-c572ab6386df", "name": "d1b4f6d3-9523-41a1-b21f-c572ab6386df", "status": "Failed", "startTime": "2026-08-06T14:06:00.450328897Z", "endTime": "2026-08-06T14:25:02.805147479Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:27:57 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-wjpjjj with version 4.21.27 Unexpected error: <fmt.wrapError \| 0xc000d24460>: failed to create HCP cluster np-version-upgrade-cluster-wjpjjj: failed waiting for cluster="np-version-upgrade-cluster-wjpjjj" in resourcegroup="rg-np-version-upgrade-wjpjjj-pc5nbpkjrzdz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/28d75c45-25ff-4278-a246-ebcc78ca1ef0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/28d75c45-25ff-4278-a246-ebcc78ca1ef0", "name": "28d75c45-25ff-4278-a246-ebcc78ca1ef0", "status": "Failed", "startTime": "2026-08-06T13:47:14.337423761Z", "endTime": "2026-08-06T14:06:32.076160168Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-wjpjjj with version 4.21.27 Unexpected error: <fmt.wrapError \| 0xc000d24460>: failed to create HCP cluster np-version-upgrade-cluster-wjpjjj: failed waiting for cluster="np-version-upgrade-cluster-wjpjjj" in resourcegroup="rg-np-version-upgrade-wjpjjj-pc5nbpkjrzdz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/28d75c45-25ff-4278-a246-ebcc78ca1ef0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/28d75c45-25ff-4278-a246-ebcc78ca1ef0", "name": "28d75c45-25ff-4278-a246-ebcc78ca1ef0", "status": "Failed", "startTime": "2026-08-06T13:47:14.337423761Z", "endTime": "2026-08-06T14:06:32.076160168Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:27:57 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:593]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc00086d0c0>: failed to create HCP cluster np-downgrade-6ff98p: failed waiting for cluster="np-downgrade-6ff98p" in resourcegroup="rg-np-downgrade-6ff98p-2wm7xthgsqdg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d67f7fc-7980-4d5a-a602-efc7b0cae3f8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d67f7fc-7980-4d5a-a602-efc7b0cae3f8", "name": "5d67f7fc-7980-4d5a-a602-efc7b0cae3f8", "status": "Failed", "startTime": "2026-08-06T14:01:51.048715965Z", "endTime": "2026-08-06T14:20:52.703469338Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:593]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc00086d0c0>: failed to create HCP cluster np-downgrade-6ff98p: failed waiting for cluster="np-downgrade-6ff98p" in resourcegroup="rg-np-downgrade-6ff98p-2wm7xthgsqdg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d67f7fc-7980-4d5a-a602-efc7b0cae3f8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d67f7fc-7980-4d5a-a602-efc7b0cae3f8", "name": "5d67f7fc-7980-4d5a-a602-efc7b0cae3f8", "status": "Failed", "startTime": "2026-08-06T14:01:51.048715965Z", "endTime": "2026-08-06T14:20:52.703469338Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:27:57 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0xc0011ea6c0>: failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-pqql6hwj4pb6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d53f51b2-3af2-44b5-a959-3b9e5b951e8f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d53f51b2-3af2-44b5-a959-3b9e5b951e8f", "name": "d53f51b2-3af2-44b5-a959-3b9e5b951e8f", "status": "Failed", "startTime": "2026-08-06T13:27:27.054348153Z", "endTime": "2026-08-06T13:46:30.524065583Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0xc0011ea6c0>: failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-pqql6hwj4pb6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d53f51b2-3af2-44b5-a959-3b9e5b951e8f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d53f51b2-3af2-44b5-a959-3b9e5b951e8f", "name": "d53f51b2-3af2-44b5-a959-3b9e5b951e8f", "status": "Failed", "startTime": "2026-08-06T13:27:27.054348153Z", "endTime": "2026-08-06T13:46:30.524065583Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:27:57 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints Unexpected error: <fmt.wrapError \| 0xc000880f00>: failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-pnf2fh5wjwql" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8925daa4-519a-42f1-a42f-eda89e216058 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8925daa4-519a-42f1-a42f-eda89e216058", "name": "8925daa4-519a-42f1-a42f-eda89e216058", "status": "Failed", "startTime": "2026-08-06T13:29:35.264600406Z", "endTime": "2026-08-06T13:48:41.617977129Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints Unexpected error: <fmt.wrapError \| 0xc000880f00>: failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-pnf2fh5wjwql" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8925daa4-519a-42f1-a42f-eda89e216058 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8925daa4-519a-42f1-a42f-eda89e216058", "name": "8925daa4-519a-42f1-a42f-eda89e216058", "status": "Failed", "startTime": "2026-08-06T13:29:35.264600406Z", "endTime": "2026-08-06T13:48:41.617977129Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift 12:27:57 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-21-ntl57w" with version 4.21.23 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000a9cd40>: failed to create HCP cluster cluster-zstream-4-21-ntl57w: failed waiting for cluster="cluster-zstream-4-21-ntl57w" in resourcegroup="rg-zstream-upgrade-4-21-tdr6mr4275wq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/00f2afbd-cf95-4467-a26f-0c9ca61e32c3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/00f2afbd-cf95-4467-a26f-0c9ca61e32c3", "name": "00f2afbd-cf95-4467-a26f-0c9ca61e32c3", "status": "Failed", "startTime": "2026-08-06T13:19:35.127061483Z", "endTime": "2026-08-06T13:38:40.310199162Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-21-ntl57w" with version 4.21.23 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000a9cd40>: failed to create HCP cluster cluster-zstream-4-21-ntl57w: failed waiting for cluster="cluster-zstream-4-21-ntl57w" in resourcegroup="rg-zstream-upgrade-4-21-tdr6mr4275wq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/00f2afbd-cf95-4467-a26f-0c9ca61e32c3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/00f2afbd-cf95-4467-a26f-0c9ca61e32c3", "name": "00f2afbd-cf95-4467-a26f-0c9ca61e32c3", "status": "Failed", "startTime": "2026-08-06T13:19:35.127061483Z", "endTime": "2026-08-06T13:38:40.310199162Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: externalAuths, nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift 12:27:57 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc001108210>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-dkkn6llwn9ml" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a058d23-b76e-4230-8a1e-4b6d1a6bf1c7 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a058d23-b76e-4230-8a1e-4b6d1a6bf1c7", "name": "6a058d23-b76e-4230-8a1e-4b6d1a6bf1c7", "status": "Failed", "startTime": "2026-08-06T13:24:37.727370739Z", "endTime": "2026-08-06T13:48:41.816178395Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s139utb7gn0spngljdld5adcoi9lknv still exists (deletion dispatched at 2026-08-06T13:24:38Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 2 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc001108210>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-dkkn6llwn9ml" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a058d23-b76e-4230-8a1e-4b6d1a6bf1c7 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a058d23-b76e-4230-8a1e-4b6d1a6bf1c7", "name": "6a058d23-b76e-4230-8a1e-4b6d1a6bf1c7", "status": "Failed", "startTime": "2026-08-06T13:24:37.727370739Z", "endTime": "2026-08-06T13:48:41.816178395Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s139utb7gn0spngljdld5adcoi9lknv still exists (deletion dispatched at 2026-08-06T13:24:38Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 2 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:27:57 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0xc0004aea40>: failed to create HCP cluster ea-listhjmvfp: failed waiting for cluster="ea-listhjmvfp" in resourcegroup="ea-list-vbbgk86jvtck" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/226ad230-01df-4367-9185-ab8b4bff928d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/226ad230-01df-4367-9185-ab8b4bff928d", "name": "226ad230-01df-4367-9185-ab8b4bff928d", "status": "Failed", "startTime": "2026-08-06T13:24:11.533675253Z", "endTime": "2026-08-06T13:43:30.417851569Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0xc0004aea40>: failed to create HCP cluster ea-listhjmvfp: failed waiting for cluster="ea-listhjmvfp" in resourcegroup="ea-list-vbbgk86jvtck" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/226ad230-01df-4367-9185-ab8b4bff928d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/226ad230-01df-4367-9185-ab8b4bff928d", "name": "226ad230-01df-4367-9185-ab8b4bff928d", "status": "Failed", "startTime": "2026-08-06T13:24:11.533675253Z", "endTime": "2026-08-06T13:43:30.417851569Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred
10:53:45	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
08:50:24	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
08:45:54	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
08:40:41	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
05:35:52	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
05:21:08	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
05:17:49	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
05:08:58	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
02:00:03	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
01:52:08	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a