CIHealth Run Log

Environment: STG

Time (UTC)	Job	Failed at	Result	PR	Failed tests	Details
23:00:11	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	32	e2e (32) ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 23:00:11 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:462]: failed to create HCP cluster np-skip-minor-v8v4bw at version 4.22.0-0.nightly-multi-2026-08-11-120309 Unexpected error: <fmt.wrapError \| 0xc000f268e0>: failed to create HCP cluster np-skip-minor-v8v4bw: failed waiting for cluster="np-skip-minor-v8v4bw" in resourcegroup="rg-np-skip-minor-v8v4bw-m9qsqfcg26w6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d8c33446-8f67-46eb-8d4f-70a6f9dd3471 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d8c33446-8f67-46eb-8d4f-70a6f9dd3471", "name": "d8c33446-8f67-46eb-8d4f-70a6f9dd3471", "status": "Failed", "startTime": "2026-08-11T23:50:01.086809244Z", "endTime": "2026-08-12T00:09:06.701618581Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:462]: failed to create HCP cluster np-skip-minor-v8v4bw at version 4.22.0-0.nightly-multi-2026-08-11-120309 Unexpected error: <fmt.wrapError \| 0xc000f268e0>: failed to create HCP cluster np-skip-minor-v8v4bw: failed waiting for cluster="np-skip-minor-v8v4bw" in resourcegroup="rg-np-skip-minor-v8v4bw-m9qsqfcg26w6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d8c33446-8f67-46eb-8d4f-70a6f9dd3471 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d8c33446-8f67-46eb-8d4f-70a6f9dd3471", "name": "d8c33446-8f67-46eb-8d4f-70a6f9dd3471", "status": "Failed", "startTime": "2026-08-11T23:50:01.086809244Z", "endTime": "2026-08-12T00:09:06.701618581Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 23:00:11 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster Unexpected error: <fmt.wrapError \| 0xc000ec72a0>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-872q6c5pdbsv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f84fe915-64a6-40a0-a041-589a090184bf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f84fe915-64a6-40a0-a041-589a090184bf", "name": "f84fe915-64a6-40a0-a041-589a090184bf", "status": "Failed", "startTime": "2026-08-11T23:36:01.387144352Z", "endTime": "2026-08-11T23:55:06.812193946Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster Unexpected error: <fmt.wrapError \| 0xc000ec72a0>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-872q6c5pdbsv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f84fe915-64a6-40a0-a041-589a090184bf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f84fe915-64a6-40a0-a041-589a090184bf", "name": "f84fe915-64a6-40a0-a041-589a090184bf", "status": "Failed", "startTime": "2026-08-11T23:36:01.387144352Z", "endTime": "2026-08-11T23:55:06.812193946Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 23:00:11 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:98]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0xc0005a0360>: failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-2295pqr4vnqk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/df22c022-84b2-42e9-b743-273877a0d7b0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/df22c022-84b2-42e9-b743-273877a0d7b0", "name": "df22c022-84b2-42e9-b743-273877a0d7b0", "status": "Failed", "startTime": "2026-08-11T23:36:06.917019748Z", "endTime": "2026-08-11T23:55:06.835167499Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:98]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0xc0005a0360>: failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-2295pqr4vnqk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/df22c022-84b2-42e9-b743-273877a0d7b0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/df22c022-84b2-42e9-b743-273877a0d7b0", "name": "df22c022-84b2-42e9-b743-273877a0d7b0", "status": "Failed", "startTime": "2026-08-11T23:36:06.917019748Z", "endTime": "2026-08-11T23:55:06.835167499Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 23:00:11 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-20-89szk8" with version 4.20.32 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000c058c0>: failed to create HCP cluster cluster-zstream-4-20-89szk8: failed waiting for cluster="cluster-zstream-4-20-89szk8" in resourcegroup="rg-zstream-upgrade-4-20-vs42d9hw65cs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/28c61d74-e56f-42f2-9428-b4ee03170bfb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/28c61d74-e56f-42f2-9428-b4ee03170bfb", "name": "28c61d74-e56f-42f2-9428-b4ee03170bfb", "status": "Failed", "startTime": "2026-08-11T23:28:06.494455039Z", "endTime": "2026-08-11T23:47:06.337628951Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-20-89szk8" with version 4.20.32 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000c058c0>: failed to create HCP cluster cluster-zstream-4-20-89szk8: failed waiting for cluster="cluster-zstream-4-20-89szk8" in resourcegroup="rg-zstream-upgrade-4-20-vs42d9hw65cs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/28c61d74-e56f-42f2-9428-b4ee03170bfb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/28c61d74-e56f-42f2-9428-b4ee03170bfb", "name": "28c61d74-e56f-42f2-9428-b4ee03170bfb", "status": "Failed", "startTime": "2026-08-11T23:28:06.494455039Z", "endTime": "2026-08-11T23:47:06.337628951Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 23:00:11 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:270]: failed to create HCP cluster shoebox-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000e16880>: failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-8c26dnrlrzdl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b99e8701-f6d9-49e5-83c7-013d841d976b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b99e8701-f6d9-49e5-83c7-013d841d976b", "name": "b99e8701-f6d9-49e5-83c7-013d841d976b", "status": "Failed", "startTime": "2026-08-11T23:50:25.381567708Z", "endTime": "2026-08-12T00:09:26.720382941Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:270]: failed to create HCP cluster shoebox-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000e16880>: failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-8c26dnrlrzdl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b99e8701-f6d9-49e5-83c7-013d841d976b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b99e8701-f6d9-49e5-83c7-013d841d976b", "name": "b99e8701-f6d9-49e5-83c7-013d841d976b", "status": "Failed", "startTime": "2026-08-11T23:50:25.381567708Z", "endTime": "2026-08-12T00:09:26.720382941Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 23:00:11 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster Unexpected error: <fmt.wrapError \| 0xc000a864e0>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-c5xnftjhfp8n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/72c120d3-7fe5-4b56-bf50-717ac085c652 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/72c120d3-7fe5-4b56-bf50-717ac085c652", "name": "72c120d3-7fe5-4b56-bf50-717ac085c652", "status": "Failed", "startTime": "2026-08-11T23:51:57.787726414Z", "endTime": "2026-08-12T00:11:06.769461941Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster Unexpected error: <fmt.wrapError \| 0xc000a864e0>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-c5xnftjhfp8n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/72c120d3-7fe5-4b56-bf50-717ac085c652 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/72c120d3-7fe5-4b56-bf50-717ac085c652", "name": "72c120d3-7fe5-4b56-bf50-717ac085c652", "status": "Failed", "startTime": "2026-08-11T23:51:57.787726414Z", "endTime": "2026-08-12T00:11:06.769461941Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 23:00:11 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.21 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-nightly-4-21-h8hmjt9f5hbv/cluster-nightly-4-21-8d2hpg should provision Unexpected error: <fmt.wrapError \| 0xc000f1ef00>: failed to create HCP cluster cluster-nightly-4-21-8d2hpg: failed waiting for cluster="cluster-nightly-4-21-8d2hpg" in resourcegroup="rg-nightly-4-21-h8hmjt9f5hbv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1f68e48e-59be-457f-827d-2ead4fee0374 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1f68e48e-59be-457f-827d-2ead4fee0374", "name": "1f68e48e-59be-457f-827d-2ead4fee0374", "status": "Failed", "startTime": "2026-08-11T23:26:46.06168874Z", "endTime": "2026-08-11T23:45:46.38773987Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-nightly-4-21-h8hmjt9f5hbv/cluster-nightly-4-21-8d2hpg should provision Unexpected error: <fmt.wrapError \| 0xc000f1ef00>: failed to create HCP cluster cluster-nightly-4-21-8d2hpg: failed waiting for cluster="cluster-nightly-4-21-8d2hpg" in resourcegroup="rg-nightly-4-21-h8hmjt9f5hbv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1f68e48e-59be-457f-827d-2ead4fee0374 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1f68e48e-59be-457f-827d-2ead4fee0374", "name": "1f68e48e-59be-457f-827d-2ead4fee0374", "status": "Failed", "startTime": "2026-08-11T23:26:46.06168874Z", "endTime": "2026-08-11T23:45:46.38773987Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift 23:00:11 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc001a061e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-gnfmr9nz8xkt" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8ba17e68-ad24-4cc5-83e9-15647fb0ca73 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8ba17e68-ad24-4cc5-83e9-15647fb0ca73", "name": "8ba17e68-ad24-4cc5-83e9-15647fb0ca73", "status": "Failed", "startTime": "2026-08-11T23:24:43.846421644Z", "endTime": "2026-08-11T23:48:46.480512763Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s4lom1h1084lkfrkvqud03tkqmhb1pn still exists (deletion dispatched at 2026-08-11T23:24:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc001a061e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-gnfmr9nz8xkt" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8ba17e68-ad24-4cc5-83e9-15647fb0ca73 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8ba17e68-ad24-4cc5-83e9-15647fb0ca73", "name": "8ba17e68-ad24-4cc5-83e9-15647fb0ca73", "status": "Failed", "startTime": "2026-08-11T23:24:43.846421644Z", "endTime": "2026-08-11T23:48:46.480512763Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s4lom1h1084lkfrkvqud03tkqmhb1pn still exists (deletion dispatched at 2026-08-11T23:24:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 23:00:11 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc0011e1278>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-6q964t878t5c" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8268305c-a869-47cb-9f39-52837fcdc35e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8268305c-a869-47cb-9f39-52837fcdc35e", "name": "8268305c-a869-47cb-9f39-52837fcdc35e", "status": "Failed", "startTime": "2026-08-11T23:29:50.798521613Z", "endTime": "2026-08-11T23:53:56.910068365Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s4lo8lv4gtesak1jnmu9b9k11fo9402 still exists (deletion dispatched at 2026-08-11T23:29:52Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc0011e1278>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-6q964t878t5c" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8268305c-a869-47cb-9f39-52837fcdc35e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8268305c-a869-47cb-9f39-52837fcdc35e", "name": "8268305c-a869-47cb-9f39-52837fcdc35e", "status": "Failed", "startTime": "2026-08-11T23:29:50.798521613Z", "endTime": "2026-08-11T23:53:56.910068365Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s4lo8lv4gtesak1jnmu9b9k11fo9402 still exists (deletion dispatched at 2026-08-11T23:29:52Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 23:00:11 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:741]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000e20c00>: failed to create HCP cluster np-dg-minor-455shf: failed waiting for cluster="np-dg-minor-455shf" in resourcegroup="rg-np-dg-minor-455shf-5bhhf9w6t5jf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e2565934-591c-460e-9dd0-16be2a7b1f38 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e2565934-591c-460e-9dd0-16be2a7b1f38", "name": "e2565934-591c-460e-9dd0-16be2a7b1f38", "status": "Failed", "startTime": "2026-08-11T23:49:53.638521938Z", "endTime": "2026-08-12T00:08:56.6357084Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:741]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000e20c00>: failed to create HCP cluster np-dg-minor-455shf: failed waiting for cluster="np-dg-minor-455shf" in resourcegroup="rg-np-dg-minor-455shf-5bhhf9w6t5jf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e2565934-591c-460e-9dd0-16be2a7b1f38 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e2565934-591c-460e-9dd0-16be2a7b1f38", "name": "e2565934-591c-460e-9dd0-16be2a7b1f38", "status": "Failed", "startTime": "2026-08-11T23:49:53.638521938Z", "endTime": "2026-08-12T00:08:56.6357084Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 23:00:11 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.0 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-nightly-5-0-hk694lqlw2bk/cluster-nightly-5-0-49kj2j should provision Unexpected error: <fmt.wrapError \| 0xc00097afe0>: failed to create HCP cluster cluster-nightly-5-0-49kj2j: failed waiting for cluster="cluster-nightly-5-0-49kj2j" in resourcegroup="rg-nightly-5-0-hk694lqlw2bk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/19e3a33f-1e44-4c92-87b9-19e748c70816 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/19e3a33f-1e44-4c92-87b9-19e748c70816", "name": "19e3a33f-1e44-4c92-87b9-19e748c70816", "status": "Failed", "startTime": "2026-08-11T23:26:58.710386207Z", "endTime": "2026-08-11T23:46:06.408061301Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-nightly-5-0-hk694lqlw2bk/cluster-nightly-5-0-49kj2j should provision Unexpected error: <fmt.wrapError \| 0xc00097afe0>: failed to create HCP cluster cluster-nightly-5-0-49kj2j: failed waiting for cluster="cluster-nightly-5-0-49kj2j" in resourcegroup="rg-nightly-5-0-hk694lqlw2bk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/19e3a33f-1e44-4c92-87b9-19e748c70816 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/19e3a33f-1e44-4c92-87b9-19e748c70816", "name": "19e3a33f-1e44-4c92-87b9-19e748c70816", "status": "Failed", "startTime": "2026-08-11T23:26:58.710386207Z", "endTime": "2026-08-11T23:46:06.408061301Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 23:00:11 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:217]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0xc000e626a0>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-rsnbnclf66qj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/39698eca-af1a-43ec-8f2f-9a71b7e5e296 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/39698eca-af1a-43ec-8f2f-9a71b7e5e296", "name": "39698eca-af1a-43ec-8f2f-9a71b7e5e296", "status": "Failed", "startTime": "2026-08-11T23:50:06.553117602Z", "endTime": "2026-08-12T00:09:06.716633972Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:217]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0xc000e626a0>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-rsnbnclf66qj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/39698eca-af1a-43ec-8f2f-9a71b7e5e296 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/39698eca-af1a-43ec-8f2f-9a71b7e5e296", "name": "39698eca-af1a-43ec-8f2f-9a71b7e5e296", "status": "Failed", "startTime": "2026-08-11T23:50:06.553117602Z", "endTime": "2026-08-12T00:09:06.716633972Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 23:00:11 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0xc000dcb420>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-2v545vp45wtx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5c52419f-fbfe-41f6-a404-3f49c8b7edaa -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5c52419f-fbfe-41f6-a404-3f49c8b7edaa", "name": "5c52419f-fbfe-41f6-a404-3f49c8b7edaa", "status": "Failed", "startTime": "2026-08-11T23:34:00.550612667Z", "endTime": "2026-08-11T23:53:06.747237705Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0xc000dcb420>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-2v545vp45wtx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5c52419f-fbfe-41f6-a404-3f49c8b7edaa -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5c52419f-fbfe-41f6-a404-3f49c8b7edaa", "name": "5c52419f-fbfe-41f6-a404-3f49c8b7edaa", "status": "Failed", "startTime": "2026-08-11T23:34:00.550612667Z", "endTime": "2026-08-11T23:53:06.747237705Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 23:00:11 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc00082fa20>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-jr9ctpwlfwjr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7091a9a5-13c6-430f-93d0-17113506a6ff -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7091a9a5-13c6-430f-93d0-17113506a6ff", "name": "7091a9a5-13c6-430f-93d0-17113506a6ff", "status": "Failed", "startTime": "2026-08-11T23:34:01.431488993Z", "endTime": "2026-08-11T23:53:06.97074243Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc00082fa20>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-jr9ctpwlfwjr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7091a9a5-13c6-430f-93d0-17113506a6ff -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7091a9a5-13c6-430f-93d0-17113506a6ff", "name": "7091a9a5-13c6-430f-93d0-17113506a6ff", "status": "Failed", "startTime": "2026-08-11T23:34:01.431488993Z", "endTime": "2026-08-11T23:53:06.97074243Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 23:00:11 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-19-r26nlz" with version 4.19.40 on candidate channel Unexpected error: <fmt.wrapError \| 0xc0004b26e0>: failed to create HCP cluster cluster-zstream-4-19-r26nlz: failed waiting for cluster="cluster-zstream-4-19-r26nlz" in resourcegroup="rg-zstream-upgrade-4-19-mrqbw599ztjb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3d85f866-fcb3-4659-93a4-75a2ad848708 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3d85f866-fcb3-4659-93a4-75a2ad848708", "name": "3d85f866-fcb3-4659-93a4-75a2ad848708", "status": "Failed", "startTime": "2026-08-11T23:28:14.107965535Z", "endTime": "2026-08-11T23:47:16.637853926Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-19-r26nlz" with version 4.19.40 on candidate channel Unexpected error: <fmt.wrapError \| 0xc0004b26e0>: failed to create HCP cluster cluster-zstream-4-19-r26nlz: failed waiting for cluster="cluster-zstream-4-19-r26nlz" in resourcegroup="rg-zstream-upgrade-4-19-mrqbw599ztjb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3d85f866-fcb3-4659-93a4-75a2ad848708 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3d85f866-fcb3-4659-93a4-75a2ad848708", "name": "3d85f866-fcb3-4659-93a4-75a2ad848708", "status": "Failed", "startTime": "2026-08-11T23:28:14.107965535Z", "endTime": "2026-08-11T23:47:16.637853926Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 23:00:11 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0xc000f8d380>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-9x8xz57k25cx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1424c2b5-7e21-4415-9741-9d6ad91e0b8d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1424c2b5-7e21-4415-9741-9d6ad91e0b8d", "name": "1424c2b5-7e21-4415-9741-9d6ad91e0b8d", "status": "Failed", "startTime": "2026-08-11T23:32:53.502571214Z", "endTime": "2026-08-11T23:51:56.487688874Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0xc000f8d380>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-9x8xz57k25cx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1424c2b5-7e21-4415-9741-9d6ad91e0b8d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1424c2b5-7e21-4415-9741-9d6ad91e0b8d", "name": "1424c2b5-7e21-4415-9741-9d6ad91e0b8d", "status": "Failed", "startTime": "2026-08-11T23:32:53.502571214Z", "endTime": "2026-08-11T23:51:56.487688874Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 23:00:11 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.22 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-nightly-4-22-2rvg98ms52gw/cluster-nightly-4-22-9ttt7x should provision Unexpected error: <fmt.wrapError \| 0xc0014141c0>: failed to create HCP cluster cluster-nightly-4-22-9ttt7x: failed waiting for cluster="cluster-nightly-4-22-9ttt7x" in resourcegroup="rg-nightly-4-22-2rvg98ms52gw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/14b29228-55ad-41bc-9514-5c375d336b03 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/14b29228-55ad-41bc-9514-5c375d336b03", "name": "14b29228-55ad-41bc-9514-5c375d336b03", "status": "Failed", "startTime": "2026-08-11T23:26:56.637572124Z", "endTime": "2026-08-11T23:45:56.116898146Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-nightly-4-22-2rvg98ms52gw/cluster-nightly-4-22-9ttt7x should provision Unexpected error: <fmt.wrapError \| 0xc0014141c0>: failed to create HCP cluster cluster-nightly-4-22-9ttt7x: failed waiting for cluster="cluster-nightly-4-22-9ttt7x" in resourcegroup="rg-nightly-4-22-2rvg98ms52gw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/14b29228-55ad-41bc-9514-5c375d336b03 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/14b29228-55ad-41bc-9514-5c375d336b03", "name": "14b29228-55ad-41bc-9514-5c375d336b03", "status": "Failed", "startTime": "2026-08-11T23:26:56.637572124Z", "endTime": "2026-08-11T23:45:56.116898146Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 23:00:11 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0xc0005a85c0>: failed to create HCP cluster ea-listtndr27: failed waiting for cluster="ea-listtndr27" in resourcegroup="ea-list-h87vt96xv6bf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5416ae06-e78d-49c0-a38d-0a968ebbca47 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5416ae06-e78d-49c0-a38d-0a968ebbca47", "name": "5416ae06-e78d-49c0-a38d-0a968ebbca47", "status": "Failed", "startTime": "2026-08-11T23:33:50.736356948Z", "endTime": "2026-08-11T23:52:56.551902262Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0xc0005a85c0>: failed to create HCP cluster ea-listtndr27: failed waiting for cluster="ea-listtndr27" in resourcegroup="ea-list-h87vt96xv6bf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5416ae06-e78d-49c0-a38d-0a968ebbca47 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5416ae06-e78d-49c0-a38d-0a968ebbca47", "name": "5416ae06-e78d-49c0-a38d-0a968ebbca47", "status": "Failed", "startTime": "2026-08-11T23:33:50.736356948Z", "endTime": "2026-08-11T23:52:56.551902262Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 23:00:11 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-21-kzxlwf" with version 4.21.27 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000fc3360>: failed to create HCP cluster cluster-zstream-4-21-kzxlwf: failed waiting for cluster="cluster-zstream-4-21-kzxlwf" in resourcegroup="rg-zstream-upgrade-4-21-ctkvqphq82sn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bd26e343-cdb7-421e-a58e-8adc8543ee23 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bd26e343-cdb7-421e-a58e-8adc8543ee23", "name": "bd26e343-cdb7-421e-a58e-8adc8543ee23", "status": "Failed", "startTime": "2026-08-11T23:30:51.25570724Z", "endTime": "2026-08-11T23:49:56.4117678Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-21-kzxlwf" with version 4.21.27 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000fc3360>: failed to create HCP cluster cluster-zstream-4-21-kzxlwf: failed waiting for cluster="cluster-zstream-4-21-kzxlwf" in resourcegroup="rg-zstream-upgrade-4-21-ctkvqphq82sn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bd26e343-cdb7-421e-a58e-8adc8543ee23 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bd26e343-cdb7-421e-a58e-8adc8543ee23", "name": "bd26e343-cdb7-421e-a58e-8adc8543ee23", "status": "Failed", "startTime": "2026-08-11T23:30:51.25570724Z", "endTime": "2026-08-11T23:49:56.4117678Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift 23:00:11 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.20 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc002c381b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-nightly-4-20-hx4vsq" in resourcegroup="rg-nightly-4-20-mlf9649gf6nw" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b353c4f6-44af-4a58-95ee-686b06d67560 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b353c4f6-44af-4a58-95ee-686b06d67560", "name": "b353c4f6-44af-4a58-95ee-686b06d67560", "status": "Failed", "startTime": "2026-08-11T23:24:36.608917138Z", "endTime": "2026-08-11T23:48:36.565035314Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s4lnvigvp182t3h8fe2d8ebnbeppjfr still exists (deletion dispatched at 2026-08-11T23:24:37Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc002c381b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-nightly-4-20-hx4vsq" in resourcegroup="rg-nightly-4-20-mlf9649gf6nw" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b353c4f6-44af-4a58-95ee-686b06d67560 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b353c4f6-44af-4a58-95ee-686b06d67560", "name": "b353c4f6-44af-4a58-95ee-686b06d67560", "status": "Failed", "startTime": "2026-08-11T23:24:36.608917138Z", "endTime": "2026-08-11T23:48:36.565035314Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s4lnvigvp182t3h8fe2d8ebnbeppjfr still exists (deletion dispatched at 2026-08-11T23:24:37Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 23:00:11 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc0000126a8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-tf5ktrtjbh5z" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1edc4471-f6a7-4bf4-9a97-0e58c926d25f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1edc4471-f6a7-4bf4-9a97-0e58c926d25f", "name": "1edc4471-f6a7-4bf4-9a97-0e58c926d25f", "status": "Failed", "startTime": "2026-08-11T23:35:18.055944742Z", "endTime": "2026-08-11T23:59:27.915516747Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s4lpbkv0a8sakjqk2n5gd31klhfru7u still exists (deletion dispatched at 2026-08-11T23:35:20Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc0000126a8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-tf5ktrtjbh5z" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1edc4471-f6a7-4bf4-9a97-0e58c926d25f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1edc4471-f6a7-4bf4-9a97-0e58c926d25f", "name": "1edc4471-f6a7-4bf4-9a97-0e58c926d25f", "status": "Failed", "startTime": "2026-08-11T23:35:18.055944742Z", "endTime": "2026-08-11T23:59:27.915516747Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s4lpbkv0a8sakjqk2n5gd31klhfru7u still exists (deletion dispatched at 2026-08-11T23:35:20Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 23:00:11 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-22-f88jxn" with version 4.22.6 on candidate channel Unexpected error: <fmt.wrapError \| 0xc00092d3a0>: failed to create HCP cluster cluster-zstream-4-22-f88jxn: failed waiting for cluster="cluster-zstream-4-22-f88jxn" in resourcegroup="rg-zstream-upgrade-4-22-p6f9z7g7lkxq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/baac4f53-12cd-4b8b-9236-ffa522b8ad35 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/baac4f53-12cd-4b8b-9236-ffa522b8ad35", "name": "baac4f53-12cd-4b8b-9236-ffa522b8ad35", "status": "Failed", "startTime": "2026-08-11T23:30:51.638118376Z", "endTime": "2026-08-11T23:49:56.400834325Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-22-f88jxn" with version 4.22.6 on candidate channel Unexpected error: <fmt.wrapError \| 0xc00092d3a0>: failed to create HCP cluster cluster-zstream-4-22-f88jxn: failed waiting for cluster="cluster-zstream-4-22-f88jxn" in resourcegroup="rg-zstream-upgrade-4-22-p6f9z7g7lkxq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/baac4f53-12cd-4b8b-9236-ffa522b8ad35 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/baac4f53-12cd-4b8b-9236-ffa522b8ad35", "name": "baac4f53-12cd-4b8b-9236-ffa522b8ad35", "status": "Failed", "startTime": "2026-08-11T23:30:51.638118376Z", "endTime": "2026-08-11T23:49:56.400834325Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 23:00:11 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints Unexpected error: <fmt.wrapError \| 0xc0008929e0>: failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-k7fpnkzrkndh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a3a5bda5-d7d0-485b-a01a-88b283314b01 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a3a5bda5-d7d0-485b-a01a-88b283314b01", "name": "a3a5bda5-d7d0-485b-a01a-88b283314b01", "status": "Failed", "startTime": "2026-08-11T23:48:31.414599631Z", "endTime": "2026-08-12T00:07:36.648446945Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints Unexpected error: <fmt.wrapError \| 0xc0008929e0>: failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-k7fpnkzrkndh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a3a5bda5-d7d0-485b-a01a-88b283314b01 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a3a5bda5-d7d0-485b-a01a-88b283314b01", "name": "a3a5bda5-d7d0-485b-a01a-88b283314b01", "status": "Failed", "startTime": "2026-08-11T23:48:31.414599631Z", "endTime": "2026-08-12T00:07:36.648446945Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 23:00:11 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.1 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-nightly-5-1-ftqqnccj9lnc/cluster-nightly-5-1-fnn2fp should provision Unexpected error: <fmt.wrapErrors \| 0xc0013eb290>: failed to create HCP cluster cluster-nightly-5-1-fnn2fp, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-1-fnn2fp in resource group rg-nightly-5-1-ftqqnccj9lnc, error: failed waiting for cluster="cluster-nightly-5-1-fnn2fp" in resourcegroup="rg-nightly-5-1-ftqqnccj9lnc" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-1-fnn2fp in resource group rg-nightly-5-1-ftqqnccj9lnc, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-nightly-5-1-ftqqnccj9lnc/cluster-nightly-5-1-fnn2fp should provision Unexpected error: <fmt.wrapErrors \| 0xc0013eb290>: failed to create HCP cluster cluster-nightly-5-1-fnn2fp, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-1-fnn2fp in resource group rg-nightly-5-1-ftqqnccj9lnc, error: failed waiting for cluster="cluster-nightly-5-1-fnn2fp" in resourcegroup="rg-nightly-5-1-ftqqnccj9lnc" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-1-fnn2fp in resource group rg-nightly-5-1-ftqqnccj9lnc, error: context deadline exceeded ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 23:00:11 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.23 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-nightly-4-23-jk26knd6fgzn/cluster-nightly-4-23-ksd282 should provision Unexpected error: <fmt.wrapError \| 0xc0007904e0>: failed to create HCP cluster cluster-nightly-4-23-ksd282: failed waiting for cluster="cluster-nightly-4-23-ksd282" in resourcegroup="rg-nightly-4-23-jk26knd6fgzn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/50d1d50b-abee-4910-b44e-270b7a53abbf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/50d1d50b-abee-4910-b44e-270b7a53abbf", "name": "50d1d50b-abee-4910-b44e-270b7a53abbf", "status": "Failed", "startTime": "2026-08-11T23:26:59.323149117Z", "endTime": "2026-08-11T23:46:06.342978979Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-nightly-4-23-jk26knd6fgzn/cluster-nightly-4-23-ksd282 should provision Unexpected error: <fmt.wrapError \| 0xc0007904e0>: failed to create HCP cluster cluster-nightly-4-23-ksd282: failed waiting for cluster="cluster-nightly-4-23-ksd282" in resourcegroup="rg-nightly-4-23-jk26knd6fgzn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/50d1d50b-abee-4910-b44e-270b7a53abbf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/50d1d50b-abee-4910-b44e-270b7a53abbf", "name": "50d1d50b-abee-4910-b44e-270b7a53abbf", "status": "Failed", "startTime": "2026-08-11T23:26:59.323149117Z", "endTime": "2026-08-11T23:46:06.342978979Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift 23:00:11 UTC · failed at e2e · test FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc001cb8270>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="fips-enabled-cluster" in resourcegroup="fips-enabled-txpstthkpqf5" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6988e3d6-531a-403f-b429-debe354d0017 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6988e3d6-531a-403f-b429-debe354d0017", "name": "6988e3d6-531a-403f-b429-debe354d0017", "status": "Failed", "startTime": "2026-08-11T23:25:10.685424571Z", "endTime": "2026-08-11T23:49:16.632334065Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s4lnu714p9rp61esar0vstdaknq7u6m still exists (deletion dispatched at 2026-08-11T23:25:11Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc001cb8270>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="fips-enabled-cluster" in resourcegroup="fips-enabled-txpstthkpqf5" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6988e3d6-531a-403f-b429-debe354d0017 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6988e3d6-531a-403f-b429-debe354d0017", "name": "6988e3d6-531a-403f-b429-debe354d0017", "status": "Failed", "startTime": "2026-08-11T23:25:10.685424571Z", "endTime": "2026-08-11T23:49:16.632334065Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s4lnu714p9rp61esar0vstdaknq7u6m still exists (deletion dispatched at 2026-08-11T23:25:11Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 23:00:11 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0xc001062920>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-zhfqgx7mngp7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/39923cf0-6127-4281-9c3c-756021b4d91a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/39923cf0-6127-4281-9c3c-756021b4d91a", "name": "39923cf0-6127-4281-9c3c-756021b4d91a", "status": "Failed", "startTime": "2026-08-11T23:35:56.77196499Z", "endTime": "2026-08-11T23:54:56.652443649Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0xc001062920>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-zhfqgx7mngp7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/39923cf0-6127-4281-9c3c-756021b4d91a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/39923cf0-6127-4281-9c3c-756021b4d91a", "name": "39923cf0-6127-4281-9c3c-756021b4d91a", "status": "Failed", "startTime": "2026-08-11T23:35:56.77196499Z", "endTime": "2026-08-11T23:54:56.652443649Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 23:00:11 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapErrors \| 0xc002c18270>: failed to create HCP cluster private-kas, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-bmlcnvz9ckpx, error: failed waiting for cluster="private-kas" in resourcegroup="private-kas-bmlcnvz9ckpx" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-bmlcnvz9ckpx, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapErrors \| 0xc002c18270>: failed to create HCP cluster private-kas, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-bmlcnvz9ckpx, error: failed waiting for cluster="private-kas" in resourcegroup="private-kas-bmlcnvz9ckpx" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-bmlcnvz9ckpx, error: context deadline exceeded ... occurred DeleteResourceGroup timed out after <minutes> minutes; context deadline exceeded 23:00:11 UTC · failed at e2e · test Customer should be able to create an HCP cluster then delete it by deleting the customer resource group · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:146]: failed to delete customer resource group "cx-rg-cluster-k6m78lg6wgqs" Unexpected error: <fmt.wrapErrors \| 0xc003849260>: failed to delete resource group, caused by: timeout '60.000000' minutes exceeded during DeleteResourceGroup for resource group cx-rg-cluster-k6m78lg6wgqs, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:146]: failed to delete customer resource group "cx-rg-cluster-k6m78lg6wgqs" Unexpected error: <fmt.wrapErrors \| 0xc003849260>: failed to delete resource group, caused by: timeout '60.000000' minutes exceeded during DeleteResourceGroup for resource group cx-rg-cluster-k6m78lg6wgqs, error: context deadline exceeded ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift 23:00:11 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc001370198>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="complex-cilium-kv-djvj7mkbzj5m" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7a598452-8f61-4a4b-80ae-f7e88c1dabdb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7a598452-8f61-4a4b-80ae-f7e88c1dabdb", "name": "7a598452-8f61-4a4b-80ae-f7e88c1dabdb", "status": "Failed", "startTime": "2026-08-11T23:26:45.856883319Z", "endTime": "2026-08-11T23:50:46.49113577Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s4lom43tvt8erhq92rpfrdqo0hb0ar9 still exists (deletion dispatched at 2026-08-11T23:26:46Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc001370198>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="complex-cilium-kv-djvj7mkbzj5m" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7a598452-8f61-4a4b-80ae-f7e88c1dabdb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7a598452-8f61-4a4b-80ae-f7e88c1dabdb", "name": "7a598452-8f61-4a4b-80ae-f7e88c1dabdb", "status": "Failed", "startTime": "2026-08-11T23:26:45.856883319Z", "endTime": "2026-08-11T23:50:46.49113577Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s4lom43tvt8erhq92rpfrdqo0hb0ar9 still exists (deletion dispatched at 2026-08-11T23:26:46Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 23:00:11 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260630preview and verify the ingress is internal · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc001c522b8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-ingress" in resourcegroup="private-ingress-g8lc5d79v57s" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/576f799b-1706-4f8f-98d7-742350600ecc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/576f799b-1706-4f8f-98d7-742350600ecc", "name": "576f799b-1706-4f8f-98d7-742350600ecc", "status": "Failed", "startTime": "2026-08-11T23:24:54.573326029Z", "endTime": "2026-08-11T23:48:56.470952982Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s4lnu3cjs9iacscf1h6mgtj3vaeb9gh still exists (deletion dispatched at 2026-08-11T23:24:55Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc001c522b8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-ingress" in resourcegroup="private-ingress-g8lc5d79v57s" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/576f799b-1706-4f8f-98d7-742350600ecc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/576f799b-1706-4f8f-98d7-742350600ecc", "name": "576f799b-1706-4f8f-98d7-742350600ecc", "status": "Failed", "startTime": "2026-08-11T23:24:54.573326029Z", "endTime": "2026-08-11T23:48:56.470952982Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s4lnu3cjs9iacscf1h6mgtj3vaeb9gh still exists (deletion dispatched at 2026-08-11T23:24:55Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift 23:00:11 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc0024d60f0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cl" in resourcegroup="cni-cilium-6p8jn7dwhrl7" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/033bebfb-a5d4-4087-b904-0f9ac77a657e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/033bebfb-a5d4-4087-b904-0f9ac77a657e", "name": "033bebfb-a5d4-4087-b904-0f9ac77a657e", "status": "Failed", "startTime": "2026-08-11T23:23:22.668117062Z", "endTime": "2026-08-11T23:47:27.276194712Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s4lnu544pda59o3s0h1v97sl8v9d5je still exists (deletion dispatched at 2026-08-11T23:23:23Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc0024d60f0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cl" in resourcegroup="cni-cilium-6p8jn7dwhrl7" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/033bebfb-a5d4-4087-b904-0f9ac77a657e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/033bebfb-a5d4-4087-b904-0f9ac77a657e", "name": "033bebfb-a5d4-4087-b904-0f9ac77a657e", "status": "Failed", "startTime": "2026-08-11T23:23:22.668117062Z", "endTime": "2026-08-11T23:47:27.276194712Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s4lnu544pda59o3s0h1v97sl8v9d5je still exists (deletion dispatched at 2026-08-11T23:23:23Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred
19:35:42	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
15:22:15	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
02:00:10	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a