CIHealth Run Log

Environment: PROD

Time (UTC)	Job	Failed at	Result	PR	Failed tests	Details
23:12:05	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	45	e2e (45) ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:312]: failed to create HCP cluster np-noedge-gcr48x Unexpected error: <fmt.wrapError \| 0xc000e687a0>: failed to create HCP cluster np-noedge-gcr48x: failed waiting for cluster="np-noedge-gcr48x" in resourcegroup="rg-np-noedge-gcr48x-zbmq2z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/14211a29-e9a6-462f-971d-3a4e45b95ee1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/14211a29-e9a6-462f-971d-3a4e45b95ee1", "name": "14211a29-e9a6-462f-971d-3a4e45b95ee1", "status": "Failed", "startTime": "2026-07-21T00:17:31.297065785Z", "endTime": "2026-07-21T00:36:37.107842263Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:312]: failed to create HCP cluster np-noedge-gcr48x Unexpected error: <fmt.wrapError \| 0xc000e687a0>: failed to create HCP cluster np-noedge-gcr48x: failed waiting for cluster="np-noedge-gcr48x" in resourcegroup="rg-np-noedge-gcr48x-zbmq2z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/14211a29-e9a6-462f-971d-3a4e45b95ee1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/14211a29-e9a6-462f-971d-3a4e45b95ee1", "name": "14211a29-e9a6-462f-971d-3a4e45b95ee1", "status": "Failed", "startTime": "2026-07-21T00:17:31.297065785Z", "endTime": "2026-07-21T00:36:37.107842263Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: failed to create HCP cluster "cluster-zstream-4-22-btk2zg" with version 4.22.4 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000f208e0>: failed to create HCP cluster cluster-zstream-4-22-btk2zg: failed waiting for cluster="cluster-zstream-4-22-btk2zg" in resourcegroup="rg-zstream-upgrade-4-22-xs4gl6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/14681ece-f5f2-45b9-8d66-0fe3b46e17ab -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/14681ece-f5f2-45b9-8d66-0fe3b46e17ab", "name": "14681ece-f5f2-45b9-8d66-0fe3b46e17ab", "status": "Failed", "startTime": "2026-07-20T23:51:50.176402512Z", "endTime": "2026-07-21T00:10:56.149266938Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: failed to create HCP cluster "cluster-zstream-4-22-btk2zg" with version 4.22.4 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000f208e0>: failed to create HCP cluster cluster-zstream-4-22-btk2zg: failed waiting for cluster="cluster-zstream-4-22-btk2zg" in resourcegroup="rg-zstream-upgrade-4-22-xs4gl6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/14681ece-f5f2-45b9-8d66-0fe3b46e17ab -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/14681ece-f5f2-45b9-8d66-0fe3b46e17ab", "name": "14681ece-f5f2-45b9-8d66-0fe3b46e17ab", "status": "Failed", "startTime": "2026-07-20T23:51:50.176402512Z", "endTime": "2026-07-21T00:10:56.149266938Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:88]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0xc000a67ae0>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-4qrkxf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5084db0d-3e6b-41e7-a0c3-5104a663ded1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5084db0d-3e6b-41e7-a0c3-5104a663ded1", "name": "5084db0d-3e6b-41e7-a0c3-5104a663ded1", "status": "Failed", "startTime": "2026-07-21T00:13:34.291625253Z", "endTime": "2026-07-21T00:32:37.715507635Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:88]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0xc000a67ae0>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-4qrkxf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5084db0d-3e6b-41e7-a0c3-5104a663ded1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5084db0d-3e6b-41e7-a0c3-5104a663ded1", "name": "5084db0d-3e6b-41e7-a0c3-5104a663ded1", "status": "Failed", "startTime": "2026-07-21T00:13:34.291625253Z", "endTime": "2026-07-21T00:32:37.715507635Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:111]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0xc000e7a620>: failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-jtl59z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4a17de45-1712-45e7-ba26-a82ec08c105e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4a17de45-1712-45e7-ba26-a82ec08c105e", "name": "4a17de45-1712-45e7-ba26-a82ec08c105e", "status": "Failed", "startTime": "2026-07-21T00:13:36.904106278Z", "endTime": "2026-07-21T00:32:37.819231401Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:111]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0xc000e7a620>: failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-jtl59z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4a17de45-1712-45e7-ba26-a82ec08c105e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4a17de45-1712-45e7-ba26-a82ec08c105e", "name": "4a17de45-1712-45e7-ba26-a82ec08c105e", "status": "Failed", "startTime": "2026-07-21T00:13:36.904106278Z", "endTime": "2026-07-21T00:32:37.819231401Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:132]: failed to create HCP cluster np-version-upgrade-cluster-lgpjr8 with version 4.21.25 Unexpected error: <fmt.wrapError \| 0xc001514a00>: failed to create HCP cluster np-version-upgrade-cluster-lgpjr8: failed waiting for cluster="np-version-upgrade-cluster-lgpjr8" in resourcegroup="rg-np-version-upgrade-lgpjr8-6kng2b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/153eb99d-6d8d-4dab-92c6-67355312b7ed -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/153eb99d-6d8d-4dab-92c6-67355312b7ed", "name": "153eb99d-6d8d-4dab-92c6-67355312b7ed", "status": "Failed", "startTime": "2026-07-21T00:36:51.729825177Z", "endTime": "2026-07-21T00:55:57.557090323Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:132]: failed to create HCP cluster np-version-upgrade-cluster-lgpjr8 with version 4.21.25 Unexpected error: <fmt.wrapError \| 0xc001514a00>: failed to create HCP cluster np-version-upgrade-cluster-lgpjr8: failed waiting for cluster="np-version-upgrade-cluster-lgpjr8" in resourcegroup="rg-np-version-upgrade-lgpjr8-6kng2b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/153eb99d-6d8d-4dab-92c6-67355312b7ed -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/153eb99d-6d8d-4dab-92c6-67355312b7ed", "name": "153eb99d-6d8d-4dab-92c6-67355312b7ed", "status": "Failed", "startTime": "2026-07-21T00:36:51.729825177Z", "endTime": "2026-07-21T00:55:57.557090323Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91]: failed to create HCP cluster cluster-np-labels-taints Unexpected error: <fmt.wrapError \| 0xc00098e220>: failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-zxn7nn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/95d9a7df-2a4e-4f84-a966-403ab91ea89a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/95d9a7df-2a4e-4f84-a966-403ab91ea89a", "name": "95d9a7df-2a4e-4f84-a966-403ab91ea89a", "status": "Failed", "startTime": "2026-07-21T00:15:37.125576763Z", "endTime": "2026-07-21T00:34:37.38346775Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91]: failed to create HCP cluster cluster-np-labels-taints Unexpected error: <fmt.wrapError \| 0xc00098e220>: failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-zxn7nn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/95d9a7df-2a4e-4f84-a966-403ab91ea89a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/95d9a7df-2a4e-4f84-a966-403ab91ea89a", "name": "95d9a7df-2a4e-4f84-a966-403ab91ea89a", "status": "Failed", "startTime": "2026-07-21T00:15:37.125576763Z", "endTime": "2026-07-21T00:34:37.38346775Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: failed to create HCP cluster "cluster-zstream-4-21-mvtdpw" with version 4.21.24 on candidate channel Unexpected error: <fmt.wrapError \| 0xc00148c160>: failed to create HCP cluster cluster-zstream-4-21-mvtdpw: failed waiting for cluster="cluster-zstream-4-21-mvtdpw" in resourcegroup="rg-zstream-upgrade-4-21-fvlw2h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/3e9e4860-a377-4798-951c-c4fb1265c36e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/3e9e4860-a377-4798-951c-c4fb1265c36e", "name": "3e9e4860-a377-4798-951c-c4fb1265c36e", "status": "Failed", "startTime": "2026-07-20T23:50:31.385489021Z", "endTime": "2026-07-21T00:09:36.292394847Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: failed to create HCP cluster "cluster-zstream-4-21-mvtdpw" with version 4.21.24 on candidate channel Unexpected error: <fmt.wrapError \| 0xc00148c160>: failed to create HCP cluster cluster-zstream-4-21-mvtdpw: failed waiting for cluster="cluster-zstream-4-21-mvtdpw" in resourcegroup="rg-zstream-upgrade-4-21-fvlw2h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/3e9e4860-a377-4798-951c-c4fb1265c36e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/3e9e4860-a377-4798-951c-c4fb1265c36e", "name": "3e9e4860-a377-4798-951c-c4fb1265c36e", "status": "Failed", "startTime": "2026-07-20T23:50:31.385489021Z", "endTime": "2026-07-21T00:09:36.292394847Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:112]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc000cc2680>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-6khn4q" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/94d788ec-189e-4362-a3d2-cec0d3cdf922 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/94d788ec-189e-4362-a3d2-cec0d3cdf922", "name": "94d788ec-189e-4362-a3d2-cec0d3cdf922", "status": "Failed", "startTime": "2026-07-20T23:53:24.790797117Z", "endTime": "2026-07-21T00:12:35.891697968Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:112]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc000cc2680>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-6khn4q" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/94d788ec-189e-4362-a3d2-cec0d3cdf922 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/94d788ec-189e-4362-a3d2-cec0d3cdf922", "name": "94d788ec-189e-4362-a3d2-cec0d3cdf922", "status": "Failed", "startTime": "2026-07-20T23:53:24.790797117Z", "endTime": "2026-07-21T00:12:35.891697968Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-4-22-zg2msw/cluster-candidate-4-22-chxf97 should provision Unexpected error: <fmt.wrapError \| 0xc000c86c80>: failed to create HCP cluster cluster-candidate-4-22-chxf97: failed waiting for cluster="cluster-candidate-4-22-chxf97" in resourcegroup="rg-candidate-4-22-zg2msw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c30d3263-35fe-4461-887e-bbf69801993f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c30d3263-35fe-4461-887e-bbf69801993f", "name": "c30d3263-35fe-4461-887e-bbf69801993f", "status": "Failed", "startTime": "2026-07-20T23:50:33.888830998Z", "endTime": "2026-07-21T00:09:35.895668016Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-4-22-zg2msw/cluster-candidate-4-22-chxf97 should provision Unexpected error: <fmt.wrapError \| 0xc000c86c80>: failed to create HCP cluster cluster-candidate-4-22-chxf97: failed waiting for cluster="cluster-candidate-4-22-chxf97" in resourcegroup="rg-candidate-4-22-zg2msw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c30d3263-35fe-4461-887e-bbf69801993f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c30d3263-35fe-4461-887e-bbf69801993f", "name": "c30d3263-35fe-4461-887e-bbf69801993f", "status": "Failed", "startTime": "2026-07-20T23:50:33.888830998Z", "endTime": "2026-07-21T00:09:35.895668016Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test FIPS Mode Support with ExperimentalReleaseFeatures AFEC registered should create an HCP cluster with FIPS mode enabled via experimental tag · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:84]: failed to create HCP cluster "fips-enabled-cluster" with FIPS enabled Unexpected error: <fmt.wrapError \| 0xc0015e0d60>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-pvvc56" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/bd62d79a-cdbf-4506-b158-5b029d4a0f93 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/bd62d79a-cdbf-4506-b158-5b029d4a0f93", "name": "bd62d79a-cdbf-4506-b158-5b029d4a0f93", "status": "Failed", "startTime": "2026-07-20T23:50:09.612127418Z", "endTime": "2026-07-21T00:09:16.010730105Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:84]: failed to create HCP cluster "fips-enabled-cluster" with FIPS enabled Unexpected error: <fmt.wrapError \| 0xc0015e0d60>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-pvvc56" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/bd62d79a-cdbf-4506-b158-5b029d4a0f93 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/bd62d79a-cdbf-4506-b158-5b029d4a0f93", "name": "bd62d79a-cdbf-4506-b158-5b029d4a0f93", "status": "Failed", "startTime": "2026-07-20T23:50:09.612127418Z", "endTime": "2026-07-21T00:09:16.010730105Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:216]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0xc000e281c0>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-p885n9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ce6ed49a-c2cd-4f6b-970a-c991b4923c0c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ce6ed49a-c2cd-4f6b-970a-c991b4923c0c", "name": "ce6ed49a-c2cd-4f6b-970a-c991b4923c0c", "status": "Failed", "startTime": "2026-07-21T00:36:43.253166262Z", "endTime": "2026-07-21T00:55:47.483495644Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:216]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0xc000e281c0>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-p885n9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ce6ed49a-c2cd-4f6b-970a-c991b4923c0c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ce6ed49a-c2cd-4f6b-970a-c991b4923c0c", "name": "ce6ed49a-c2cd-4f6b-970a-c991b4923c0c", "status": "Failed", "startTime": "2026-07-21T00:36:43.253166262Z", "endTime": "2026-07-21T00:55:47.483495644Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: failed to create HCP cluster "cluster-zstream-4-20-dkbjrb" with version 4.20.29 on candidate channel Unexpected error: <fmt.wrapError \| 0xc00098a860>: failed to create HCP cluster cluster-zstream-4-20-dkbjrb: failed waiting for cluster="cluster-zstream-4-20-dkbjrb" in resourcegroup="rg-zstream-upgrade-4-20-2nmjxs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/da2e092d-f95f-491f-a903-7f3eea4bbef8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/da2e092d-f95f-491f-a903-7f3eea4bbef8", "name": "da2e092d-f95f-491f-a903-7f3eea4bbef8", "status": "Failed", "startTime": "2026-07-20T23:50:35.181363652Z", "endTime": "2026-07-21T00:09:35.893935187Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: failed to create HCP cluster "cluster-zstream-4-20-dkbjrb" with version 4.20.29 on candidate channel Unexpected error: <fmt.wrapError \| 0xc00098a860>: failed to create HCP cluster cluster-zstream-4-20-dkbjrb: failed waiting for cluster="cluster-zstream-4-20-dkbjrb" in resourcegroup="rg-zstream-upgrade-4-20-2nmjxs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/da2e092d-f95f-491f-a903-7f3eea4bbef8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/da2e092d-f95f-491f-a903-7f3eea4bbef8", "name": "da2e092d-f95f-491f-a903-7f3eea4bbef8", "status": "Failed", "startTime": "2026-07-20T23:50:35.181363652Z", "endTime": "2026-07-21T00:09:35.893935187Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:83]: failed to create HCP cluster "cilium-cl" without CNI Unexpected error: <fmt.wrapError \| 0xc000c73aa0>: failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-wbrpm7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/810fd8dc-75cc-46e7-9935-40b1c132cd65 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/810fd8dc-75cc-46e7-9935-40b1c132cd65", "name": "810fd8dc-75cc-46e7-9935-40b1c132cd65", "status": "Failed", "startTime": "2026-07-20T23:27:20.299583025Z", "endTime": "2026-07-20T23:46:34.265720908Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:83]: failed to create HCP cluster "cilium-cl" without CNI Unexpected error: <fmt.wrapError \| 0xc000c73aa0>: failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-wbrpm7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/810fd8dc-75cc-46e7-9935-40b1c132cd65 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/810fd8dc-75cc-46e7-9935-40b1c132cd65", "name": "810fd8dc-75cc-46e7-9935-40b1c132cd65", "status": "Failed", "startTime": "2026-07-20T23:27:20.299583025Z", "endTime": "2026-07-20T23:46:34.265720908Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-4-20-jx4gz7/cluster-candidate-4-20-7xw8dx should provision Unexpected error: <fmt.wrapError \| 0xc000c5f000>: failed to create HCP cluster cluster-candidate-4-20-7xw8dx: failed waiting for cluster="cluster-candidate-4-20-7xw8dx" in resourcegroup="rg-candidate-4-20-jx4gz7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8aefc8b8-8aca-4959-92ef-4a3aabb46376 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8aefc8b8-8aca-4959-92ef-4a3aabb46376", "name": "8aefc8b8-8aca-4959-92ef-4a3aabb46376", "status": "Failed", "startTime": "2026-07-20T23:52:12.967428063Z", "endTime": "2026-07-21T00:11:16.713261936Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-4-20-jx4gz7/cluster-candidate-4-20-7xw8dx should provision Unexpected error: <fmt.wrapError \| 0xc000c5f000>: failed to create HCP cluster cluster-candidate-4-20-7xw8dx: failed waiting for cluster="cluster-candidate-4-20-7xw8dx" in resourcegroup="rg-candidate-4-20-jx4gz7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8aefc8b8-8aca-4959-92ef-4a3aabb46376 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8aefc8b8-8aca-4959-92ef-4a3aabb46376", "name": "8aefc8b8-8aca-4959-92ef-4a3aabb46376", "status": "Failed", "startTime": "2026-07-20T23:52:12.967428063Z", "endTime": "2026-07-21T00:11:16.713261936Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:132]: failed to create HCP cluster np-version-upgrade-cluster-7d4pkq with version 4.21.25 Unexpected error: <fmt.wrapError \| 0xc00046d700>: failed to create HCP cluster np-version-upgrade-cluster-7d4pkq: failed waiting for cluster="np-version-upgrade-cluster-7d4pkq" in resourcegroup="rg-np-version-upgrade-7d4pkq-v625bd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7733647b-dd05-400c-8ba7-8196eeb49794 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7733647b-dd05-400c-8ba7-8196eeb49794", "name": "7733647b-dd05-400c-8ba7-8196eeb49794", "status": "Failed", "startTime": "2026-07-21T00:17:39.55384181Z", "endTime": "2026-07-21T00:36:57.108773159Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:132]: failed to create HCP cluster np-version-upgrade-cluster-7d4pkq with version 4.21.25 Unexpected error: <fmt.wrapError \| 0xc00046d700>: failed to create HCP cluster np-version-upgrade-cluster-7d4pkq: failed waiting for cluster="np-version-upgrade-cluster-7d4pkq" in resourcegroup="rg-np-version-upgrade-7d4pkq-v625bd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7733647b-dd05-400c-8ba7-8196eeb49794 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7733647b-dd05-400c-8ba7-8196eeb49794", "name": "7733647b-dd05-400c-8ba7-8196eeb49794", "status": "Failed", "startTime": "2026-07-21T00:17:39.55384181Z", "endTime": "2026-07-21T00:36:57.108773159Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:132]: failed to create HCP cluster np-version-upgrade-cluster-6clfvk with version 4.20.30 Unexpected error: <fmt.wrapError \| 0xc000b823e0>: failed to create HCP cluster np-version-upgrade-cluster-6clfvk: failed waiting for cluster="np-version-upgrade-cluster-6clfvk" in resourcegroup="rg-np-version-upgrade-6clfvk-bbn9b6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4b55f620-ed87-4313-b1bb-6323b9bf092a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4b55f620-ed87-4313-b1bb-6323b9bf092a", "name": "4b55f620-ed87-4313-b1bb-6323b9bf092a", "status": "Failed", "startTime": "2026-07-21T00:13:39.321054413Z", "endTime": "2026-07-21T00:32:47.18921824Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:132]: failed to create HCP cluster np-version-upgrade-cluster-6clfvk with version 4.20.30 Unexpected error: <fmt.wrapError \| 0xc000b823e0>: failed to create HCP cluster np-version-upgrade-cluster-6clfvk: failed waiting for cluster="np-version-upgrade-cluster-6clfvk" in resourcegroup="rg-np-version-upgrade-6clfvk-bbn9b6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4b55f620-ed87-4313-b1bb-6323b9bf092a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4b55f620-ed87-4313-b1bb-6323b9bf092a", "name": "4b55f620-ed87-4313-b1bb-6323b9bf092a", "status": "Failed", "startTime": "2026-07-21T00:13:39.321054413Z", "endTime": "2026-07-21T00:32:47.18921824Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:86]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0xc000a22980>: failed to create HCP cluster ea-listmz5z29: failed waiting for cluster="ea-listmz5z29" in resourcegroup="ea-list-mlwjs9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0dbffcd5-faaa-4add-82fb-7df60b416cce -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0dbffcd5-faaa-4add-82fb-7df60b416cce", "name": "0dbffcd5-faaa-4add-82fb-7df60b416cce", "status": "Failed", "startTime": "2026-07-20T23:53:09.285772804Z", "endTime": "2026-07-21T00:12:16.021816887Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:86]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0xc000a22980>: failed to create HCP cluster ea-listmz5z29: failed waiting for cluster="ea-listmz5z29" in resourcegroup="ea-list-mlwjs9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0dbffcd5-faaa-4add-82fb-7df60b416cce -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0dbffcd5-faaa-4add-82fb-7df60b416cce", "name": "0dbffcd5-faaa-4add-82fb-7df60b416cce", "status": "Failed", "startTime": "2026-07-20T23:53:09.285772804Z", "endTime": "2026-07-21T00:12:16.021816887Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:109]: failed to create HCP cluster negative-tests-cluster Unexpected error: <fmt.wrapError \| 0xc000706440>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-zvmc8f" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/988235b5-720f-487c-8072-3201187fba27 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/988235b5-720f-487c-8072-3201187fba27", "name": "988235b5-720f-487c-8072-3201187fba27", "status": "Failed", "startTime": "2026-07-21T00:16:25.886659508Z", "endTime": "2026-07-21T00:35:37.084005503Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:109]: failed to create HCP cluster negative-tests-cluster Unexpected error: <fmt.wrapError \| 0xc000706440>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-zvmc8f" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/988235b5-720f-487c-8072-3201187fba27 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/988235b5-720f-487c-8072-3201187fba27", "name": "988235b5-720f-487c-8072-3201187fba27", "status": "Failed", "startTime": "2026-07-21T00:16:25.886659508Z", "endTime": "2026-07-21T00:35:37.084005503Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:736]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000f56c60>: failed to create HCP cluster np-dg-minor-948grk: failed waiting for cluster="np-dg-minor-948grk" in resourcegroup="rg-np-dg-minor-948grk-d2ndbs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5dd650d3-6e39-4696-be62-516c9def9199 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5dd650d3-6e39-4696-be62-516c9def9199", "name": "5dd650d3-6e39-4696-be62-516c9def9199", "status": "Failed", "startTime": "2026-07-21T00:15:37.246546157Z", "endTime": "2026-07-21T00:34:37.47529642Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:736]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000f56c60>: failed to create HCP cluster np-dg-minor-948grk: failed waiting for cluster="np-dg-minor-948grk" in resourcegroup="rg-np-dg-minor-948grk-d2ndbs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5dd650d3-6e39-4696-be62-516c9def9199 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5dd650d3-6e39-4696-be62-516c9def9199", "name": "5dd650d3-6e39-4696-be62-516c9def9199", "status": "Failed", "startTime": "2026-07-21T00:15:37.246546157Z", "endTime": "2026-07-21T00:34:37.47529642Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 23:12:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: failed to create HCP cluster version 4.19 Unexpected error: <fmt.wrapErrors \| 0xc000fee390>: failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-d2dn29" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-d2dn29, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: failed to create HCP cluster version 4.19 Unexpected error: <fmt.wrapErrors \| 0xc000fee390>: failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-d2dn29" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-d2dn29, error: context deadline exceeded ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: failed to create HCP cluster "cluster-zstream-4-19-9tdk5f" with version 4.19.37 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000de1580>: failed to create HCP cluster cluster-zstream-4-19-9tdk5f: failed waiting for cluster="cluster-zstream-4-19-9tdk5f" in resourcegroup="rg-zstream-upgrade-4-19-5xdq9f" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/9a4a0278-31ba-43d1-9cce-357ca5cf2db9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/9a4a0278-31ba-43d1-9cce-357ca5cf2db9", "name": "9a4a0278-31ba-43d1-9cce-357ca5cf2db9", "status": "Failed", "startTime": "2026-07-20T23:50:24.532800656Z", "endTime": "2026-07-21T00:09:35.869682594Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: failed to create HCP cluster "cluster-zstream-4-19-9tdk5f" with version 4.19.37 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000de1580>: failed to create HCP cluster cluster-zstream-4-19-9tdk5f: failed waiting for cluster="cluster-zstream-4-19-9tdk5f" in resourcegroup="rg-zstream-upgrade-4-19-5xdq9f" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/9a4a0278-31ba-43d1-9cce-357ca5cf2db9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/9a4a0278-31ba-43d1-9cce-357ca5cf2db9", "name": "9a4a0278-31ba-43d1-9cce-357ca5cf2db9", "status": "Failed", "startTime": "2026-07-20T23:50:24.532800656Z", "endTime": "2026-07-21T00:09:35.869682594Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidTemplateDeployment; provider Microsoft.Network 23:12:05 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:131]: failed to deploy test VM Unexpected error: <fmt.wrapError \| 0xc00155ef00>: failed creating deployment "test-vm" in resourcegroup="e2e-cidr-connectivity-l96fjf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-l96fjf/providers/Microsoft.Resources/deployments/test-vm -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidTemplateDeployment -------------------------------------------------------------------------------- { "error": { "code": "InvalidTemplateDeployment", "message": "The template deployment 'test-vm' is not valid according to the validation procedure. The following resource provider(s) - 'Microsoft.Network/publicIPAddresses (2022-07-01)' reported preflight validation errors. Tracking id is '3a84a790-e152-4d73-a198-bf0f9d6b6d4a'. See inner errors for details.", "details": [ { "code": "ResourceCountExceedsLimitDueToTemplate", "message": "Subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has a quota of 300 for resources of type PublicIpAddress with sku SkuNotSpecified. Subscription currently has 300 resources and the template contains 1 new resources of the this type which exceeds the quota. Please contact support to increase the quota for resource type PublicIpAddress", "details": [] } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:131]: failed to deploy test VM Unexpected error: <fmt.wrapError \| 0xc00155ef00>: failed creating deployment "test-vm" in resourcegroup="e2e-cidr-connectivity-l96fjf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-l96fjf/providers/Microsoft.Resources/deployments/test-vm -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidTemplateDeployment -------------------------------------------------------------------------------- { "error": { "code": "InvalidTemplateDeployment", "message": "The template deployment 'test-vm' is not valid according to the validation procedure. The following resource provider(s) - 'Microsoft.Network/publicIPAddresses (2022-07-01)' reported preflight validation errors. Tracking id is '3a84a790-e152-4d73-a198-bf0f9d6b6d4a'. See inner errors for details.", "details": [ { "code": "ResourceCountExceedsLimitDueToTemplate", "message": "Subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has a quota of 300 for resources of type PublicIpAddress with sku SkuNotSpecified. Subscription currently has 300 resources and the template contains 1 new resources of the this type which exceeds the quota. Please contact support to increase the quota for resource type PublicIpAddress", "details": [] } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:289]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0xc000cb8300>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-jztjj4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/47296f8d-600b-40e8-b68e-81a15af3b0a5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/47296f8d-600b-40e8-b68e-81a15af3b0a5", "name": "47296f8d-600b-40e8-b68e-81a15af3b0a5", "status": "Failed", "startTime": "2026-07-21T00:13:25.428084669Z", "endTime": "2026-07-21T00:32:37.0533164Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:289]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0xc000cb8300>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-jztjj4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/47296f8d-600b-40e8-b68e-81a15af3b0a5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/47296f8d-600b-40e8-b68e-81a15af3b0a5", "name": "47296f8d-600b-40e8-b68e-81a15af3b0a5", "status": "Failed", "startTime": "2026-07-21T00:13:25.428084669Z", "endTime": "2026-07-21T00:32:37.0533164Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0xc000e28ca0>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-dxs24s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e6e549d7-9327-473c-85da-efe60d6b427d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e6e549d7-9327-473c-85da-efe60d6b427d", "name": "e6e549d7-9327-473c-85da-efe60d6b427d", "status": "Failed", "startTime": "2026-07-20T23:51:44.983259602Z", "endTime": "2026-07-21T00:10:56.043526282Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0xc000e28ca0>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-dxs24s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e6e549d7-9327-473c-85da-efe60d6b427d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e6e549d7-9327-473c-85da-efe60d6b427d", "name": "e6e549d7-9327-473c-85da-efe60d6b427d", "status": "Failed", "startTime": "2026-07-20T23:51:44.983259602Z", "endTime": "2026-07-21T00:10:56.043526282Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:589]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000e086a0>: failed to create HCP cluster np-downgrade-kfxhrl: failed waiting for cluster="np-downgrade-kfxhrl" in resourcegroup="rg-np-downgrade-kfxhrl-2h659z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fae6bfc3-5e71-499a-93d1-a47be79e943a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fae6bfc3-5e71-499a-93d1-a47be79e943a", "name": "fae6bfc3-5e71-499a-93d1-a47be79e943a", "status": "Failed", "startTime": "2026-07-21T00:15:07.341508774Z", "endTime": "2026-07-21T00:34:07.168232803Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:589]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000e086a0>: failed to create HCP cluster np-downgrade-kfxhrl: failed waiting for cluster="np-downgrade-kfxhrl" in resourcegroup="rg-np-downgrade-kfxhrl-2h659z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fae6bfc3-5e71-499a-93d1-a47be79e943a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fae6bfc3-5e71-499a-93d1-a47be79e943a", "name": "fae6bfc3-5e71-499a-93d1-a47be79e943a", "status": "Failed", "startTime": "2026-07-21T00:15:07.341508774Z", "endTime": "2026-07-21T00:34:07.168232803Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:83]: failed to create HCP cluster for patch-name test Unexpected error: <fmt.wrapError \| 0xc000f1eba0>: failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-6vq7tp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/05a64bb6-f871-4528-ac47-c25653cada42 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/05a64bb6-f871-4528-ac47-c25653cada42", "name": "05a64bb6-f871-4528-ac47-c25653cada42", "status": "Failed", "startTime": "2026-07-20T23:27:20.134166987Z", "endTime": "2026-07-20T23:46:34.547971528Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:83]: failed to create HCP cluster for patch-name test Unexpected error: <fmt.wrapError \| 0xc000f1eba0>: failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-6vq7tp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/05a64bb6-f871-4528-ac47-c25653cada42 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/05a64bb6-f871-4528-ac47-c25653cada42", "name": "05a64bb6-f871-4528-ac47-c25653cada42", "status": "Failed", "startTime": "2026-07-20T23:27:20.134166987Z", "endTime": "2026-07-20T23:46:34.547971528Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred cluster entered terminal Failed state after role assignment deployment 23:12:05 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:219]: Timed out after 2520.001s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:215 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:219]: Timed out after 2520.001s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:215 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '20.000000' minutes) Unexpected error: <exported.ResponseError \| 0xc00024eb10>: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fc5f5768-125a-45ca-a834-7a33b15f8899 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fc5f5768-125a-45ca-a834-7a33b15f8899", "name": "fc5f5768-125a-45ca-a834-7a33b15f8899", "status": "Failed", "startTime": "2026-07-20T23:28:31.724021061Z", "endTime": "2026-07-20T23:47:34.550898973Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '20.000000' minutes) Unexpected error: <exported.ResponseError \| 0xc00024eb10>: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fc5f5768-125a-45ca-a834-7a33b15f8899 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fc5f5768-125a-45ca-a834-7a33b15f8899", "name": "fc5f5768-125a-45ca-a834-7a33b15f8899", "status": "Failed", "startTime": "2026-07-20T23:28:31.724021061Z", "endTime": "2026-07-20T23:47:34.550898973Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster then delete it by deleting the customer resource group · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:89]: failed to create HCP cluster "cx-rg-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000a36220>: failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-svbm77" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/2b4cc035-2489-4402-8e4b-b27e3e569b77 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/2b4cc035-2489-4402-8e4b-b27e3e569b77", "name": "2b4cc035-2489-4402-8e4b-b27e3e569b77", "status": "Failed", "startTime": "2026-07-20T23:54:26.376607075Z", "endTime": "2026-07-21T00:13:35.916969396Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:89]: failed to create HCP cluster "cx-rg-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000a36220>: failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-svbm77" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/2b4cc035-2489-4402-8e4b-b27e3e569b77 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/2b4cc035-2489-4402-8e4b-b27e3e569b77", "name": "2b4cc035-2489-4402-8e4b-b27e3e569b77", "status": "Failed", "startTime": "2026-07-20T23:54:26.376607075Z", "endTime": "2026-07-21T00:13:35.916969396Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:79]: failed to create HCP cluster "hcp-cluster-np-128" Unexpected error: <fmt.wrapError \| 0xc000a95080>: failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-284lhg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1bf09854-b364-4307-aa58-06efc6669be2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1bf09854-b364-4307-aa58-06efc6669be2", "name": "1bf09854-b364-4307-aa58-06efc6669be2", "status": "Failed", "startTime": "2026-07-20T23:27:19.686753629Z", "endTime": "2026-07-20T23:46:34.53808567Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:79]: failed to create HCP cluster "hcp-cluster-np-128" Unexpected error: <fmt.wrapError \| 0xc000a95080>: failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-284lhg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1bf09854-b364-4307-aa58-06efc6669be2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1bf09854-b364-4307-aa58-06efc6669be2", "name": "1bf09854-b364-4307-aa58-06efc6669be2", "status": "Failed", "startTime": "2026-07-20T23:27:19.686753629Z", "endTime": "2026-07-20T23:46:34.53808567Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred Cluster provisioning failed 23:12:05 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:157]: Cluster provisioning failed fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:157]: Cluster provisioning failed ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:97]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc0009c29c0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-gm2sxx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7ab4f41d-5636-49c8-bc7d-221e53fd1451 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7ab4f41d-5636-49c8-bc7d-221e53fd1451", "name": "7ab4f41d-5636-49c8-bc7d-221e53fd1451", "status": "Failed", "startTime": "2026-07-21T00:15:51.339800606Z", "endTime": "2026-07-21T00:35:07.082540317Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:97]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc0009c29c0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-gm2sxx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7ab4f41d-5636-49c8-bc7d-221e53fd1451 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7ab4f41d-5636-49c8-bc7d-221e53fd1451", "name": "7ab4f41d-5636-49c8-bc7d-221e53fd1451", "status": "Failed", "startTime": "2026-07-21T00:15:51.339800606Z", "endTime": "2026-07-21T00:35:07.082540317Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling Unexpected error: <fmt.wrapError \| 0xc0009016c0>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-57pvp7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/68731108-ecd6-4d35-91ce-45f888679665 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/68731108-ecd6-4d35-91ce-45f888679665", "name": "68731108-ecd6-4d35-91ce-45f888679665", "status": "Failed", "startTime": "2026-07-20T23:27:18.798916736Z", "endTime": "2026-07-20T23:46:34.278050405Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling Unexpected error: <fmt.wrapError \| 0xc0009016c0>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-57pvp7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/68731108-ecd6-4d35-91ce-45f888679665 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/68731108-ecd6-4d35-91ce-45f888679665", "name": "68731108-ecd6-4d35-91ce-45f888679665", "status": "Failed", "startTime": "2026-07-20T23:27:18.798916736Z", "endTime": "2026-07-20T23:46:34.278050405Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:269]: failed to create HCP cluster shoebox-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000d9f3c0>: failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-47xxkw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8c0c597e-e571-4665-8ebb-68ebd42a574e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8c0c597e-e571-4665-8ebb-68ebd42a574e", "name": "8c0c597e-e571-4665-8ebb-68ebd42a574e", "status": "Failed", "startTime": "2026-07-21T00:16:04.344384294Z", "endTime": "2026-07-21T00:35:07.067692277Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:269]: failed to create HCP cluster shoebox-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000d9f3c0>: failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-47xxkw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8c0c597e-e571-4665-8ebb-68ebd42a574e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8c0c597e-e571-4665-8ebb-68ebd42a574e", "name": "8c0c597e-e571-4665-8ebb-68ebd42a574e", "status": "Failed", "startTime": "2026-07-21T00:16:04.344384294Z", "endTime": "2026-07-21T00:35:07.067692277Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:104]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc000e22240>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-k4rlzd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/69814de1-7bfc-4593-85ab-d74e6cad3ef4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/69814de1-7bfc-4593-85ab-d74e6cad3ef4", "name": "69814de1-7bfc-4593-85ab-d74e6cad3ef4", "status": "Failed", "startTime": "2026-07-20T23:29:03.175561473Z", "endTime": "2026-07-20T23:48:14.452637462Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:104]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc000e22240>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-k4rlzd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/69814de1-7bfc-4593-85ab-d74e6cad3ef4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/69814de1-7bfc-4593-85ab-d74e6cad3ef4", "name": "69814de1-7bfc-4593-85ab-d74e6cad3ef4", "status": "Failed", "startTime": "2026-07-20T23:29:03.175561473Z", "endTime": "2026-07-20T23:48:14.452637462Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:101]: failed to create HCP cluster np-autoscale-cluster Unexpected error: <fmt.wrapError \| 0xc001928380>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-p2zzff" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/60f51ae9-cccd-4523-aabf-8a2f7e591bab -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/60f51ae9-cccd-4523-aabf-8a2f7e591bab", "name": "60f51ae9-cccd-4523-aabf-8a2f7e591bab", "status": "Failed", "startTime": "2026-07-21T00:13:21.077768725Z", "endTime": "2026-07-21T00:32:37.33599024Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:101]: failed to create HCP cluster np-autoscale-cluster Unexpected error: <fmt.wrapError \| 0xc001928380>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-p2zzff" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/60f51ae9-cccd-4523-aabf-8a2f7e591bab -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/60f51ae9-cccd-4523-aabf-8a2f7e591bab", "name": "60f51ae9-cccd-4523-aabf-8a2f7e591bab", "status": "Failed", "startTime": "2026-07-21T00:13:21.077768725Z", "endTime": "2026-07-21T00:32:37.33599024Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test Unexpected error: <fmt.wrapError \| 0xc000e264c0>: failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-xfqmmd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/90f81d48-56f6-4384-930a-ed2bd730dc6f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/90f81d48-56f6-4384-930a-ed2bd730dc6f", "name": "90f81d48-56f6-4384-930a-ed2bd730dc6f", "status": "Failed", "startTime": "2026-07-20T23:27:21.997343947Z", "endTime": "2026-07-20T23:46:34.293460852Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test Unexpected error: <fmt.wrapError \| 0xc000e264c0>: failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-xfqmmd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/90f81d48-56f6-4384-930a-ed2bd730dc6f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/90f81d48-56f6-4384-930a-ed2bd730dc6f", "name": "90f81d48-56f6-4384-930a-ed2bd730dc6f", "status": "Failed", "startTime": "2026-07-20T23:27:21.997343947Z", "endTime": "2026-07-20T23:46:34.293460852Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:181]: failed to create HCP cluster "cp-ystream-upgrade-4-21-w5htc7" with install version 4.20.30 Unexpected error: <fmt.wrapError \| 0xc0009f21a0>: failed to create HCP cluster cp-ystream-upgrade-4-21-w5htc7: failed waiting for cluster="cp-ystream-upgrade-4-21-w5htc7" in resourcegroup="rg-cp-ystream-upgrade-4-21-pmtzgk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7f61abdb-048c-4804-bef0-d3df10817907 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7f61abdb-048c-4804-bef0-d3df10817907", "name": "7f61abdb-048c-4804-bef0-d3df10817907", "status": "Failed", "startTime": "2026-07-20T23:51:17.625121238Z", "endTime": "2026-07-21T00:10:36.475521937Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:181]: failed to create HCP cluster "cp-ystream-upgrade-4-21-w5htc7" with install version 4.20.30 Unexpected error: <fmt.wrapError \| 0xc0009f21a0>: failed to create HCP cluster cp-ystream-upgrade-4-21-w5htc7: failed waiting for cluster="cp-ystream-upgrade-4-21-w5htc7" in resourcegroup="rg-cp-ystream-upgrade-4-21-pmtzgk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7f61abdb-048c-4804-bef0-d3df10817907 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7f61abdb-048c-4804-bef0-d3df10817907", "name": "7f61abdb-048c-4804-bef0-d3df10817907", "status": "Failed", "startTime": "2026-07-20T23:51:17.625121238Z", "endTime": "2026-07-21T00:10:36.475521937Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-5-0-956kml/cluster-candidate-5-0-kp9g8t should provision Unexpected error: <fmt.wrapError \| 0xc000e98aa0>: failed to create HCP cluster cluster-candidate-5-0-kp9g8t: failed waiting for cluster="cluster-candidate-5-0-kp9g8t" in resourcegroup="rg-candidate-5-0-956kml" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4a55d137-4efa-4047-a368-8b3e575e0eec -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4a55d137-4efa-4047-a368-8b3e575e0eec", "name": "4a55d137-4efa-4047-a368-8b3e575e0eec", "status": "Failed", "startTime": "2026-07-20T23:50:25.613264438Z", "endTime": "2026-07-21T00:09:36.076795552Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-5-0-956kml/cluster-candidate-5-0-kp9g8t should provision Unexpected error: <fmt.wrapError \| 0xc000e98aa0>: failed to create HCP cluster cluster-candidate-5-0-kp9g8t: failed waiting for cluster="cluster-candidate-5-0-kp9g8t" in resourcegroup="rg-candidate-5-0-956kml" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4a55d137-4efa-4047-a368-8b3e575e0eec -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4a55d137-4efa-4047-a368-8b3e575e0eec", "name": "4a55d137-4efa-4047-a368-8b3e575e0eec", "status": "Failed", "startTime": "2026-07-20T23:50:25.613264438Z", "endTime": "2026-07-21T00:09:36.076795552Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create HCP cluster np-skip-minor-xg2hgt at version 4.22.6 Unexpected error: <fmt.wrapError \| 0xc000a9ed40>: failed to create HCP cluster np-skip-minor-xg2hgt: failed waiting for cluster="np-skip-minor-xg2hgt" in resourcegroup="rg-np-skip-minor-xg2hgt-xnlznk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0ebbb47f-36c2-4b3e-9a5b-a1f27da4b9ef -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0ebbb47f-36c2-4b3e-9a5b-a1f27da4b9ef", "name": "0ebbb47f-36c2-4b3e-9a5b-a1f27da4b9ef", "status": "Failed", "startTime": "2026-07-21T00:14:06.59397692Z", "endTime": "2026-07-21T00:33:07.499260544Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create HCP cluster np-skip-minor-xg2hgt at version 4.22.6 Unexpected error: <fmt.wrapError \| 0xc000a9ed40>: failed to create HCP cluster np-skip-minor-xg2hgt: failed waiting for cluster="np-skip-minor-xg2hgt" in resourcegroup="rg-np-skip-minor-xg2hgt-xnlznk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0ebbb47f-36c2-4b3e-9a5b-a1f27da4b9ef -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0ebbb47f-36c2-4b3e-9a5b-a1f27da4b9ef", "name": "0ebbb47f-36c2-4b3e-9a5b-a1f27da4b9ef", "status": "Failed", "startTime": "2026-07-21T00:14:06.59397692Z", "endTime": "2026-07-21T00:33:07.499260544Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:94]: failed to create HCP cluster for TLS endpoint test Unexpected error: <fmt.wrapError \| 0xc000c44180>: failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-5w6k65" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/cd38e039-4112-483b-8393-033cdc84a47d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/cd38e039-4112-483b-8393-033cdc84a47d", "name": "cd38e039-4112-483b-8393-033cdc84a47d", "status": "Failed", "startTime": "2026-07-20T23:27:16.161321606Z", "endTime": "2026-07-20T23:46:34.29581269Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:94]: failed to create HCP cluster for TLS endpoint test Unexpected error: <fmt.wrapError \| 0xc000c44180>: failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-5w6k65" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/cd38e039-4112-483b-8393-033cdc84a47d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/cd38e039-4112-483b-8393-033cdc84a47d", "name": "cd38e039-4112-483b-8393-033cdc84a47d", "status": "Failed", "startTime": "2026-07-20T23:27:16.161321606Z", "endTime": "2026-07-20T23:46:34.29581269Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-4-21-pt2hjb/cluster-candidate-4-21-2ll7bn should provision Unexpected error: <fmt.wrapError \| 0xc000d0a820>: failed to create HCP cluster cluster-candidate-4-21-2ll7bn: failed waiting for cluster="cluster-candidate-4-21-2ll7bn" in resourcegroup="rg-candidate-4-21-pt2hjb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7fc635fe-f006-451b-aefc-b2ffc01fb9b8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7fc635fe-f006-451b-aefc-b2ffc01fb9b8", "name": "7fc635fe-f006-451b-aefc-b2ffc01fb9b8", "status": "Failed", "startTime": "2026-07-20T23:50:10.418970082Z", "endTime": "2026-07-21T00:09:16.325493793Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-4-21-pt2hjb/cluster-candidate-4-21-2ll7bn should provision Unexpected error: <fmt.wrapError \| 0xc000d0a820>: failed to create HCP cluster cluster-candidate-4-21-2ll7bn: failed waiting for cluster="cluster-candidate-4-21-2ll7bn" in resourcegroup="rg-candidate-4-21-pt2hjb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7fc635fe-f006-451b-aefc-b2ffc01fb9b8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7fc635fe-f006-451b-aefc-b2ffc01fb9b8", "name": "7fc635fe-f006-451b-aefc-b2ffc01fb9b8", "status": "Failed", "startTime": "2026-07-20T23:50:10.418970082Z", "endTime": "2026-07-21T00:09:16.325493793Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster Unexpected error: <fmt.wrapError \| 0xc001484a00>: failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-7qdfln" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b10c0065-7db4-441a-b557-aecb0bf8c76e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b10c0065-7db4-441a-b557-aecb0bf8c76e", "name": "b10c0065-7db4-441a-b557-aecb0bf8c76e", "status": "Failed", "startTime": "2026-07-20T23:27:21.322451152Z", "endTime": "2026-07-20T23:46:34.544383871Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster Unexpected error: <fmt.wrapError \| 0xc001484a00>: failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-7qdfln" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b10c0065-7db4-441a-b557-aecb0bf8c76e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b10c0065-7db4-441a-b557-aecb0bf8c76e", "name": "b10c0065-7db4-441a-b557-aecb0bf8c76e", "status": "Failed", "startTime": "2026-07-20T23:27:21.322451152Z", "endTime": "2026-07-20T23:46:34.544383871Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:87]: failed to create HCP cluster "arm64-vm-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000db82a0>: failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-n542fg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/580fc606-4c6a-421a-85bd-2417037c3c7b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/580fc606-4c6a-421a-85bd-2417037c3c7b", "name": "580fc606-4c6a-421a-85bd-2417037c3c7b", "status": "Failed", "startTime": "2026-07-20T23:30:13.298295724Z", "endTime": "2026-07-20T23:49:14.484408565Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:87]: failed to create HCP cluster "arm64-vm-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000db82a0>: failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-n542fg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/580fc606-4c6a-421a-85bd-2417037c3c7b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/580fc606-4c6a-421a-85bd-2417037c3c7b", "name": "580fc606-4c6a-421a-85bd-2417037c3c7b", "status": "Failed", "startTime": "2026-07-20T23:30:13.298295724Z", "endTime": "2026-07-20T23:49:14.484408565Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 23:12:05 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:159]: failed to create HCP cluster for patch-tags test Unexpected error: <fmt.wrapError \| 0xc000c80980>: failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-wn7gpr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1f4cf303-79f1-4123-a109-479a19aeed98 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1f4cf303-79f1-4123-a109-479a19aeed98", "name": "1f4cf303-79f1-4123-a109-479a19aeed98", "status": "Failed", "startTime": "2026-07-20T23:50:17.685484658Z", "endTime": "2026-07-21T00:09:36.162887848Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:159]: failed to create HCP cluster for patch-tags test Unexpected error: <fmt.wrapError \| 0xc000c80980>: failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-wn7gpr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1f4cf303-79f1-4123-a109-479a19aeed98 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1f4cf303-79f1-4123-a109-479a19aeed98", "name": "1f4cf303-79f1-4123-a109-479a19aeed98", "status": "Failed", "startTime": "2026-07-20T23:50:17.685484658Z", "endTime": "2026-07-21T00:09:36.162887848Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred
23:00:44	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	1	e2e (1) timeout during CreateHCPClusterAndWait; context deadline exceeded 23:00:44 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.0 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-nightly-5-0-cpmsgk/cluster-nightly-5-0-vxt256 should provision Unexpected error: <fmt.wrapErrors \| 0xc00189f830>: failed to create HCP cluster cluster-nightly-5-0-vxt256, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-0-vxt256 in resource group rg-nightly-5-0-cpmsgk, error: failed waiting for cluster="cluster-nightly-5-0-vxt256" in resourcegroup="rg-nightly-5-0-cpmsgk" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-0-vxt256 in resource group rg-nightly-5-0-cpmsgk, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-nightly-5-0-cpmsgk/cluster-nightly-5-0-vxt256 should provision Unexpected error: <fmt.wrapErrors \| 0xc00189f830>: failed to create HCP cluster cluster-nightly-5-0-vxt256, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-0-vxt256 in resource group rg-nightly-5-0-cpmsgk, error: failed waiting for cluster="cluster-nightly-5-0-vxt256" in resourcegroup="rg-nightly-5-0-cpmsgk" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-0-vxt256 in resource group rg-nightly-5-0-cpmsgk, error: context deadline exceeded ... occurred
21:27:14	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	1	e2e (1) tls: failed to verify certificate: x509: certificate signed by unknown authority 21:27:14 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:170]: failed to run simple web app and connectivity check app with cilium CNI Unexpected error: <errors.joinError \| 0xc001a0b110>: VerifySimpleWebApp failed: route was never reachable: Get "https://agnhost-e2e-sample-app-7gqvj.apps.aro.cilium-cl.gb92.westeurope.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:170]: failed to run simple web app and connectivity check app with cilium CNI Unexpected error: <errors.joinError \| 0xc001a0b110>: VerifySimpleWebApp failed: route was never reachable: Get "https://agnhost-e2e-sample-app-7gqvj.apps.aro.cilium-cl.gb92.westeurope.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority ... occurred
21:24:13	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
17:07:03	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
16:50:42	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
16:02:17	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	45	e2e (45) ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:94]: failed to create HCP cluster for TLS endpoint test Unexpected error: <fmt.wrapError \| 0xc0010301a0>: failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-hfrj7l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/edef3ced-f4ba-4b55-ac59-e8f085ddfaf0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/edef3ced-f4ba-4b55-ac59-e8f085ddfaf0", "name": "edef3ced-f4ba-4b55-ac59-e8f085ddfaf0", "status": "Failed", "startTime": "2026-07-20T16:18:15.649450825Z", "endTime": "2026-07-20T16:37:17.0362895Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:94]: failed to create HCP cluster for TLS endpoint test Unexpected error: <fmt.wrapError \| 0xc0010301a0>: failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-hfrj7l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/edef3ced-f4ba-4b55-ac59-e8f085ddfaf0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/edef3ced-f4ba-4b55-ac59-e8f085ddfaf0", "name": "edef3ced-f4ba-4b55-ac59-e8f085ddfaf0", "status": "Failed", "startTime": "2026-07-20T16:18:15.649450825Z", "endTime": "2026-07-20T16:37:17.0362895Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:97]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000fd41a0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-g4bhdc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7edda124-19b7-473c-9887-19f43cdb8aea -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7edda124-19b7-473c-9887-19f43cdb8aea", "name": "7edda124-19b7-473c-9887-19f43cdb8aea", "status": "Failed", "startTime": "2026-07-20T16:20:10.545223447Z", "endTime": "2026-07-20T16:39:16.751711284Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:97]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000fd41a0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-g4bhdc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7edda124-19b7-473c-9887-19f43cdb8aea -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7edda124-19b7-473c-9887-19f43cdb8aea", "name": "7edda124-19b7-473c-9887-19f43cdb8aea", "status": "Failed", "startTime": "2026-07-20T16:20:10.545223447Z", "endTime": "2026-07-20T16:39:16.751711284Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:736]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000b8ae40>: failed to create HCP cluster np-dg-minor-5flzzg: failed waiting for cluster="np-dg-minor-5flzzg" in resourcegroup="rg-np-dg-minor-5flzzg-pckfx9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c0849783-5217-4af9-ae68-8d3950320c0e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c0849783-5217-4af9-ae68-8d3950320c0e", "name": "c0849783-5217-4af9-ae68-8d3950320c0e", "status": "Failed", "startTime": "2026-07-20T17:51:45.816351028Z", "endTime": "2026-07-20T18:10:49.661294684Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:736]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000b8ae40>: failed to create HCP cluster np-dg-minor-5flzzg: failed waiting for cluster="np-dg-minor-5flzzg" in resourcegroup="rg-np-dg-minor-5flzzg-pckfx9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c0849783-5217-4af9-ae68-8d3950320c0e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c0849783-5217-4af9-ae68-8d3950320c0e", "name": "c0849783-5217-4af9-ae68-8d3950320c0e", "status": "Failed", "startTime": "2026-07-20T17:51:45.816351028Z", "endTime": "2026-07-20T18:10:49.661294684Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '20.000000' minutes) Unexpected error: <exported.ResponseError \| 0xc0010e6cf0>: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/365c8668-5b79-47ca-a36e-146e244d8ba6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/365c8668-5b79-47ca-a36e-146e244d8ba6", "name": "365c8668-5b79-47ca-a36e-146e244d8ba6", "status": "Failed", "startTime": "2026-07-20T16:19:19.453873297Z", "endTime": "2026-07-20T16:38:26.729684039Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '20.000000' minutes) Unexpected error: <exported.ResponseError \| 0xc0010e6cf0>: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/365c8668-5b79-47ca-a36e-146e244d8ba6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/365c8668-5b79-47ca-a36e-146e244d8ba6", "name": "365c8668-5b79-47ca-a36e-146e244d8ba6", "status": "Failed", "startTime": "2026-07-20T16:19:19.453873297Z", "endTime": "2026-07-20T16:38:26.729684039Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:87]: failed to create HCP cluster "arm64-vm-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc00080a520>: failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-zbssbt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/277e8a14-8fa1-4d41-ba83-33fbe3ee356d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/277e8a14-8fa1-4d41-ba83-33fbe3ee356d", "name": "277e8a14-8fa1-4d41-ba83-33fbe3ee356d", "status": "Failed", "startTime": "2026-07-20T16:18:04.688858269Z", "endTime": "2026-07-20T16:37:17.040207063Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:87]: failed to create HCP cluster "arm64-vm-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc00080a520>: failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-zbssbt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/277e8a14-8fa1-4d41-ba83-33fbe3ee356d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/277e8a14-8fa1-4d41-ba83-33fbe3ee356d", "name": "277e8a14-8fa1-4d41-ba83-33fbe3ee356d", "status": "Failed", "startTime": "2026-07-20T16:18:04.688858269Z", "endTime": "2026-07-20T16:37:17.040207063Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: failed to create HCP cluster "cluster-zstream-4-20-6dlm44" with version 4.20.29 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000e9c180>: failed to create HCP cluster cluster-zstream-4-20-6dlm44: failed waiting for cluster="cluster-zstream-4-20-6dlm44" in resourcegroup="rg-zstream-upgrade-4-20-j4ptp9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/dc12fe55-3a9b-4a1f-8c88-2777eae6961c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/dc12fe55-3a9b-4a1f-8c88-2777eae6961c", "name": "dc12fe55-3a9b-4a1f-8c88-2777eae6961c", "status": "Failed", "startTime": "2026-07-20T17:05:49.652398803Z", "endTime": "2026-07-20T17:24:58.42709892Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: failed to create HCP cluster "cluster-zstream-4-20-6dlm44" with version 4.20.29 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000e9c180>: failed to create HCP cluster cluster-zstream-4-20-6dlm44: failed waiting for cluster="cluster-zstream-4-20-6dlm44" in resourcegroup="rg-zstream-upgrade-4-20-j4ptp9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/dc12fe55-3a9b-4a1f-8c88-2777eae6961c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/dc12fe55-3a9b-4a1f-8c88-2777eae6961c", "name": "dc12fe55-3a9b-4a1f-8c88-2777eae6961c", "status": "Failed", "startTime": "2026-07-20T17:05:49.652398803Z", "endTime": "2026-07-20T17:24:58.42709892Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred Deserialization Error: no output from command 16:02:17 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel Show raw failure 053c60, 0xc0004e9b00}, {{0xc000f8eec0, 0x1, 0x1}, 0x34630b8a000, {0x871a24d, 0x8}, {{0x0, ...}, ...}}) /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:371 +0x252 fp=0xc00190f730 sp=0xc00190f618 pc=0x6c9d292 github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).deleteCreatedResources(0xc000bc26e0, {0x7f2434053c60, 0xc0004e9b00}) /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:262 +0x250 fp=0xc00190f888 sp=0xc00190f730 pc=0x6c9c9f0 github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).deleteCreatedResources-fm({0x7f2434053c60?, 0xc0004e9b00?}) <autogenerated>:1 +0x33 fp=0xc00190f8b0 sp=0xc00190f888 pc=0x6cbcad3 runtime.call16(0xc0004bf380, 0xc0006f95c0, 0x0, 0x0, 0x0, 0x10, 0xc00190fb08) /usr/lib/golang/src/runtime/asm_amd64.s:774 +0x43 fp=0xc00190f8d0 sp=0xc00190f8b0 pc=0x2f2a223 runtime.reflectcall(0x80e5200?, 0xc0004e9b00?, 0x2?, 0x8751e1d?, 0x0?, 0x12?, 0x7c9d5e0?) <autogenerated>:1 +0x36 fp=0xc00190f910 sp=0xc00190f8d0 pc=0x2f32ab6 reflect.Value.call({0x75465a0?, 0xc0006f95c0?, 0xc000d70d20?}, {0x870dbd9, 0x4}, {0xc000c24f60, 0x1, 0x0?}) /usr/lib/golang/src/reflect/value.go:581 +0xcc6 fp=0xc00190fe70 sp=0xc00190f910 pc=0x2f65c66 reflect.Value.Call({0x75465a0?, 0xc0006f95c0?, 0xc000d70d20?}, {0xc000c24f60?, 0xc000bc26e0?, 0x932c968?}) /usr/lib/golang/src/reflect/value.go:365 +0xb9 fp=0xc00190fee8 sp=0xc00190fe70 pc=0x2f64d59 github.com/onsi/ginkgo/v2/internal.NewCleanupNode.func2({0x93472c8?, 0xc0004e9b00?}) /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:589 +0x150 fp=0xc00190ff88 sp=0xc00190fee8 pc=0x3388c70 github.com/onsi/ginkgo/v2/internal.(Suite).runNode.func3() /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc00190ffe0 sp=0xc00190ff88 pc=0x339b73b runtime.goexit({}) /usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00190ffe8 sp=0xc00190ffe0 pc=0x2f2bcc1 created by github.com/onsi/ginkgo/v2/internal.(Suite).runNode in goroutine 120 /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50 goroutine 377 gp=0xc0016fa1c0 m=nil [select]: runtime.gopark(0xc000d5d900?, 0x2?, 0x0?, 0xbe?, 0xc000d5d8f4?) /usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000d5d780 sp=0xc000d5d760 pc=0x2f2350e runtime.selectgo(0xc000d5d900, 0xc000d5d8f0, 0xc0015dec80?, 0x0, 0x36f17dc?, 0x1) /usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc000d5d8c0 sp=0xc000d5d780 pc=0x2efe2f7 github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x932c9d8, 0xc00071cbd0}, 0x8775b21?) /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/internal/shared/shared.go:36 +0x76 fp=0xc000d5d930 sp=0xc000d5d8c0 pc=0x36f0f36 github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(Poller[...]).PollUntilDone(0x9377d20, {0x932c9d8, 0xc00071cbd0}, 0x11) /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/runtime/poller.go:284 +0x732 fp=0xc000d5dc30 sp=0xc000d5d930 pc=0x3dfc6f2 github.com/Azure/ARO-HCP/test/util/framework.DeleteResourceGroup({0x7f2434053c60, 0xc0004e9b00}, 0xc00154a960, 0xc000cc2de0, {0xc000c6ce10, 0x11}, 0x0, 0x34630b8a000) /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/resourcegroup_helper.go:172 +0x6aa fp=0xc000d5dd90 sp=0xc000d5dc30 pc=0x6ca92ca github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).cleanupResourceGroup(0xc000bc26e0, {0x7f2434053c60, 0xc0004e9b00}, {0xc000c6ce10, 0x11}, 0x34630b8a000) /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:568 +0x8a5 fp=0xc000d5df08 sp=0xc000d5dd90 pc=0x6c9fd05 github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).CleanupResourceGroups.func1({0x7f2434053c60?, 0xc0004e9b00?}) /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:360 +0x151 fp=0xc000d5dfc0 sp=0xc000d5df08 pc=0x6c9d791 github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).CleanupResourceGroups.gowrap1() /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:369 +0x28 fp=0xc000d5dfe0 sp=0xc000d5dfc0 pc=0x6c9d608 runtime.goexit({}) /usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000d5dfe8 sp=0xc000d5dfe0 pc=0x2f2bcc1 created by github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).CleanupResourceGroups in goroutine 420 /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353 +0x98 rax 0xca rbx 0x0 rcx 0x2f2dae3 rdx 0x0 rdi 0xc9599f0 rsi 0x80 rbp 0x7ffc3cc57520 rsp 0x7ffc3cc574d8 r8 0x0 r9 0x0 r10 0x0 r11 0x286 r12 0x7 r13 0x1 r14 0xc957640 r15 0x7 rip 0x2f2dae1 rflags 0x286 cs 0x33 fs 0x0 gs 0x0 Command Error: exit status 2 Deserialization Error: no output from command 053c60, 0xc0004e9b00}, {{0xc000f8eec0, 0x1, 0x1}, 0x34630b8a000, {0x871a24d, 0x8}, {{0x0, ...}, ...}}) /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:371 +0x252 fp=0xc00190f730 sp=0xc00190f618 pc=0x6c9d292 github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).deleteCreatedResources(0xc000bc26e0, {0x7f2434053c60, 0xc0004e9b00}) /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:262 +0x250 fp=0xc00190f888 sp=0xc00190f730 pc=0x6c9c9f0 github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).deleteCreatedResources-fm({0x7f2434053c60?, 0xc0004e9b00?}) <autogenerated>:1 +0x33 fp=0xc00190f8b0 sp=0xc00190f888 pc=0x6cbcad3 runtime.call16(0xc0004bf380, 0xc0006f95c0, 0x0, 0x0, 0x0, 0x10, 0xc00190fb08) /usr/lib/golang/src/runtime/asm_amd64.s:774 +0x43 fp=0xc00190f8d0 sp=0xc00190f8b0 pc=0x2f2a223 runtime.reflectcall(0x80e5200?, 0xc0004e9b00?, 0x2?, 0x8751e1d?, 0x0?, 0x12?, 0x7c9d5e0?) <autogenerated>:1 +0x36 fp=0xc00190f910 sp=0xc00190f8d0 pc=0x2f32ab6 reflect.Value.call({0x75465a0?, 0xc0006f95c0?, 0xc000d70d20?}, {0x870dbd9, 0x4}, {0xc000c24f60, 0x1, 0x0?}) /usr/lib/golang/src/reflect/value.go:581 +0xcc6 fp=0xc00190fe70 sp=0xc00190f910 pc=0x2f65c66 reflect.Value.Call({0x75465a0?, 0xc0006f95c0?, 0xc000d70d20?}, {0xc000c24f60?, 0xc000bc26e0?, 0x932c968?}) /usr/lib/golang/src/reflect/value.go:365 +0xb9 fp=0xc00190fee8 sp=0xc00190fe70 pc=0x2f64d59 github.com/onsi/ginkgo/v2/internal.NewCleanupNode.func2({0x93472c8?, 0xc0004e9b00?}) /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:589 +0x150 fp=0xc00190ff88 sp=0xc00190fee8 pc=0x3388c70 github.com/onsi/ginkgo/v2/internal.(Suite).runNode.func3() /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc00190ffe0 sp=0xc00190ff88 pc=0x339b73b runtime.goexit({}) /usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00190ffe8 sp=0xc00190ffe0 pc=0x2f2bcc1 created by github.com/onsi/ginkgo/v2/internal.(Suite).runNode in goroutine 120 /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50 goroutine 377 gp=0xc0016fa1c0 m=nil [select]: runtime.gopark(0xc000d5d900?, 0x2?, 0x0?, 0xbe?, 0xc000d5d8f4?) /usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000d5d780 sp=0xc000d5d760 pc=0x2f2350e runtime.selectgo(0xc000d5d900, 0xc000d5d8f0, 0xc0015dec80?, 0x0, 0x36f17dc?, 0x1) /usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc000d5d8c0 sp=0xc000d5d780 pc=0x2efe2f7 github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x932c9d8, 0xc00071cbd0}, 0x8775b21?) /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/internal/shared/shared.go:36 +0x76 fp=0xc000d5d930 sp=0xc000d5d8c0 pc=0x36f0f36 github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(Poller[...]).PollUntilDone(0x9377d20, {0x932c9d8, 0xc00071cbd0}, 0x11) /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/runtime/poller.go:284 +0x732 fp=0xc000d5dc30 sp=0xc000d5d930 pc=0x3dfc6f2 github.com/Azure/ARO-HCP/test/util/framework.DeleteResourceGroup({0x7f2434053c60, 0xc0004e9b00}, 0xc00154a960, 0xc000cc2de0, {0xc000c6ce10, 0x11}, 0x0, 0x34630b8a000) /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/resourcegroup_helper.go:172 +0x6aa fp=0xc000d5dd90 sp=0xc000d5dc30 pc=0x6ca92ca github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).cleanupResourceGroup(0xc000bc26e0, {0x7f2434053c60, 0xc0004e9b00}, {0xc000c6ce10, 0x11}, 0x34630b8a000) /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:568 +0x8a5 fp=0xc000d5df08 sp=0xc000d5dd90 pc=0x6c9fd05 github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).CleanupResourceGroups.func1({0x7f2434053c60?, 0xc0004e9b00?}) /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:360 +0x151 fp=0xc000d5dfc0 sp=0xc000d5df08 pc=0x6c9d791 github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).CleanupResourceGroups.gowrap1() /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:369 +0x28 fp=0xc000d5dfe0 sp=0xc000d5dfc0 pc=0x6c9d608 runtime.goexit({}) /usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000d5dfe8 sp=0xc000d5dfe0 pc=0x2f2bcc1 created by github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).CleanupResourceGroups in goroutine 420 /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353 +0x98 rax 0xca rbx 0x0 rcx 0x2f2dae3 rdx 0x0 rdi 0xc9599f0 rsi 0x80 rbp 0x7ffc3cc57520 rsp 0x7ffc3cc574d8 r8 0x0 r9 0x0 r10 0x0 r11 0x286 r12 0x7 r13 0x1 r14 0xc957640 r15 0x7 rip 0x2f2dae1 rflags 0x286 cs 0x33 fs 0x0 gs 0x0 Command Error: exit status 2 Deserialization Error: no output from command ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-4-21-hjh4kk/cluster-candidate-4-21-65t9r9 should provision Unexpected error: <fmt.wrapError \| 0xc00092c340>: failed to create HCP cluster cluster-candidate-4-21-65t9r9: failed waiting for cluster="cluster-candidate-4-21-65t9r9" in resourcegroup="rg-candidate-4-21-hjh4kk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/3d5c1eb6-ed8a-4a20-b4e0-f916d47db593 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/3d5c1eb6-ed8a-4a20-b4e0-f916d47db593", "name": "3d5c1eb6-ed8a-4a20-b4e0-f916d47db593", "status": "Failed", "startTime": "2026-07-20T17:04:00.634426013Z", "endTime": "2026-07-20T17:23:18.533587615Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-4-21-hjh4kk/cluster-candidate-4-21-65t9r9 should provision Unexpected error: <fmt.wrapError \| 0xc00092c340>: failed to create HCP cluster cluster-candidate-4-21-65t9r9: failed waiting for cluster="cluster-candidate-4-21-65t9r9" in resourcegroup="rg-candidate-4-21-hjh4kk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/3d5c1eb6-ed8a-4a20-b4e0-f916d47db593 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/3d5c1eb6-ed8a-4a20-b4e0-f916d47db593", "name": "3d5c1eb6-ed8a-4a20-b4e0-f916d47db593", "status": "Failed", "startTime": "2026-07-20T17:04:00.634426013Z", "endTime": "2026-07-20T17:23:18.533587615Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster Unexpected error: <fmt.wrapError \| 0xc000f9cea0>: failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-wrhqrn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7c043db6-4547-4dff-8663-cbbb45a7f09b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7c043db6-4547-4dff-8663-cbbb45a7f09b", "name": "7c043db6-4547-4dff-8663-cbbb45a7f09b", "status": "Failed", "startTime": "2026-07-20T16:17:54.986152146Z", "endTime": "2026-07-20T16:36:57.261502092Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster Unexpected error: <fmt.wrapError \| 0xc000f9cea0>: failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-wrhqrn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7c043db6-4547-4dff-8663-cbbb45a7f09b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7c043db6-4547-4dff-8663-cbbb45a7f09b", "name": "7c043db6-4547-4dff-8663-cbbb45a7f09b", "status": "Failed", "startTime": "2026-07-20T16:17:54.986152146Z", "endTime": "2026-07-20T16:36:57.261502092Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:312]: failed to create HCP cluster np-noedge-bqqkqf Unexpected error: <fmt.wrapError \| 0xc000e66620>: failed to create HCP cluster np-noedge-bqqkqf: failed waiting for cluster="np-noedge-bqqkqf" in resourcegroup="rg-np-noedge-bqqkqf-h9dwjk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/537156bb-ec75-4dfc-9eaa-2e613bfeccc5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/537156bb-ec75-4dfc-9eaa-2e613bfeccc5", "name": "537156bb-ec75-4dfc-9eaa-2e613bfeccc5", "status": "Failed", "startTime": "2026-07-20T17:51:23.596727955Z", "endTime": "2026-07-20T18:10:39.449977043Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:312]: failed to create HCP cluster np-noedge-bqqkqf Unexpected error: <fmt.wrapError \| 0xc000e66620>: failed to create HCP cluster np-noedge-bqqkqf: failed waiting for cluster="np-noedge-bqqkqf" in resourcegroup="rg-np-noedge-bqqkqf-h9dwjk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/537156bb-ec75-4dfc-9eaa-2e613bfeccc5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/537156bb-ec75-4dfc-9eaa-2e613bfeccc5", "name": "537156bb-ec75-4dfc-9eaa-2e613bfeccc5", "status": "Failed", "startTime": "2026-07-20T17:51:23.596727955Z", "endTime": "2026-07-20T18:10:39.449977043Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:132]: failed to create HCP cluster np-version-upgrade-cluster-pkpwnj with version 4.21.25 Unexpected error: <fmt.wrapError \| 0xc00069a5c0>: failed to create HCP cluster np-version-upgrade-cluster-pkpwnj: failed waiting for cluster="np-version-upgrade-cluster-pkpwnj" in resourcegroup="rg-np-version-upgrade-pkpwnj-k52f24" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/a5eafffd-12ed-47f9-9d60-036543f6b686 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/a5eafffd-12ed-47f9-9d60-036543f6b686", "name": "a5eafffd-12ed-47f9-9d60-036543f6b686", "status": "Failed", "startTime": "2026-07-20T17:51:00.884238157Z", "endTime": "2026-07-20T18:10:19.645069551Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:132]: failed to create HCP cluster np-version-upgrade-cluster-pkpwnj with version 4.21.25 Unexpected error: <fmt.wrapError \| 0xc00069a5c0>: failed to create HCP cluster np-version-upgrade-cluster-pkpwnj: failed waiting for cluster="np-version-upgrade-cluster-pkpwnj" in resourcegroup="rg-np-version-upgrade-pkpwnj-k52f24" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/a5eafffd-12ed-47f9-9d60-036543f6b686 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/a5eafffd-12ed-47f9-9d60-036543f6b686", "name": "a5eafffd-12ed-47f9-9d60-036543f6b686", "status": "Failed", "startTime": "2026-07-20T17:51:00.884238157Z", "endTime": "2026-07-20T18:10:19.645069551Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Customer should be able to create an HCP cluster then delete it by deleting the customer resource group · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:89]: failed to create HCP cluster "cx-rg-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000f1a540>: failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-sqpj6p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6d995035-f3d7-41a6-a77e-fa47789e448e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6d995035-f3d7-41a6-a77e-fa47789e448e", "name": "6d995035-f3d7-41a6-a77e-fa47789e448e", "status": "Failed", "startTime": "2026-07-20T16:18:07.28543095Z", "endTime": "2026-07-20T16:37:17.106236321Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:89]: failed to create HCP cluster "cx-rg-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000f1a540>: failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-sqpj6p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6d995035-f3d7-41a6-a77e-fa47789e448e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6d995035-f3d7-41a6-a77e-fa47789e448e", "name": "6d995035-f3d7-41a6-a77e-fa47789e448e", "status": "Failed", "startTime": "2026-07-20T16:18:07.28543095Z", "endTime": "2026-07-20T16:37:17.106236321Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:88]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0xc000c50f00>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-wb96pq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b074b0ac-92c8-47eb-a70e-6beb167049ae -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b074b0ac-92c8-47eb-a70e-6beb167049ae", "name": "b074b0ac-92c8-47eb-a70e-6beb167049ae", "status": "Failed", "startTime": "2026-07-20T17:08:03.365199196Z", "endTime": "2026-07-20T17:27:18.710801787Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:88]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0xc000c50f00>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-wb96pq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b074b0ac-92c8-47eb-a70e-6beb167049ae -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b074b0ac-92c8-47eb-a70e-6beb167049ae", "name": "b074b0ac-92c8-47eb-a70e-6beb167049ae", "status": "Failed", "startTime": "2026-07-20T17:08:03.365199196Z", "endTime": "2026-07-20T17:27:18.710801787Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:79]: failed to create HCP cluster "hcp-cluster-np-128" Unexpected error: <fmt.wrapError \| 0xc000dac6a0>: failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-jbhkmb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0af36a75-ce36-4b0c-96da-437af5e4a9cc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0af36a75-ce36-4b0c-96da-437af5e4a9cc", "name": "0af36a75-ce36-4b0c-96da-437af5e4a9cc", "status": "Failed", "startTime": "2026-07-20T16:18:00.987915077Z", "endTime": "2026-07-20T16:37:16.685766584Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:79]: failed to create HCP cluster "hcp-cluster-np-128" Unexpected error: <fmt.wrapError \| 0xc000dac6a0>: failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-jbhkmb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0af36a75-ce36-4b0c-96da-437af5e4a9cc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0af36a75-ce36-4b0c-96da-437af5e4a9cc", "name": "0af36a75-ce36-4b0c-96da-437af5e4a9cc", "status": "Failed", "startTime": "2026-07-20T16:18:00.987915077Z", "endTime": "2026-07-20T16:37:16.685766584Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0xc000952d80>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-q8znls" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/87ceb102-b0ec-4c82-980b-84d79a51deef -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/87ceb102-b0ec-4c82-980b-84d79a51deef", "name": "87ceb102-b0ec-4c82-980b-84d79a51deef", "status": "Failed", "startTime": "2026-07-20T17:05:40.800964096Z", "endTime": "2026-07-20T17:24:48.200930424Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0xc000952d80>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-q8znls" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/87ceb102-b0ec-4c82-980b-84d79a51deef -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/87ceb102-b0ec-4c82-980b-84d79a51deef", "name": "87ceb102-b0ec-4c82-980b-84d79a51deef", "status": "Failed", "startTime": "2026-07-20T17:05:40.800964096Z", "endTime": "2026-07-20T17:24:48.200930424Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:83]: failed to create HCP cluster for patch-name test Unexpected error: <fmt.wrapError \| 0xc000110200>: failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-2qqmns" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/20db9755-b153-4bb2-8706-0c0a4c914345 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/20db9755-b153-4bb2-8706-0c0a4c914345", "name": "20db9755-b153-4bb2-8706-0c0a4c914345", "status": "Failed", "startTime": "2026-07-20T17:06:08.019702048Z", "endTime": "2026-07-20T17:25:18.208249507Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:83]: failed to create HCP cluster for patch-name test Unexpected error: <fmt.wrapError \| 0xc000110200>: failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-2qqmns" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/20db9755-b153-4bb2-8706-0c0a4c914345 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/20db9755-b153-4bb2-8706-0c0a4c914345", "name": "20db9755-b153-4bb2-8706-0c0a4c914345", "status": "Failed", "startTime": "2026-07-20T17:06:08.019702048Z", "endTime": "2026-07-20T17:25:18.208249507Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:83]: failed to create HCP cluster "cilium-cl" without CNI Unexpected error: <fmt.wrapError \| 0xc000e9c560>: failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-dx4wm6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ef24c0d4-f3ad-4935-8711-e4bc4f11c7dc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ef24c0d4-f3ad-4935-8711-e4bc4f11c7dc", "name": "ef24c0d4-f3ad-4935-8711-e4bc4f11c7dc", "status": "Failed", "startTime": "2026-07-20T16:18:06.065322164Z", "endTime": "2026-07-20T16:37:16.698553789Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:83]: failed to create HCP cluster "cilium-cl" without CNI Unexpected error: <fmt.wrapError \| 0xc000e9c560>: failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-dx4wm6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ef24c0d4-f3ad-4935-8711-e4bc4f11c7dc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ef24c0d4-f3ad-4935-8711-e4bc4f11c7dc", "name": "ef24c0d4-f3ad-4935-8711-e4bc4f11c7dc", "status": "Failed", "startTime": "2026-07-20T16:18:06.065322164Z", "endTime": "2026-07-20T16:37:16.698553789Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:112]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc0011eaba0>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-6z844q" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/471255a0-9ea3-4828-a373-c34d03625965 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/471255a0-9ea3-4828-a373-c34d03625965", "name": "471255a0-9ea3-4828-a373-c34d03625965", "status": "Failed", "startTime": "2026-07-20T17:07:06.252833555Z", "endTime": "2026-07-20T17:26:18.713895489Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:112]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc0011eaba0>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-6z844q" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/471255a0-9ea3-4828-a373-c34d03625965 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/471255a0-9ea3-4828-a373-c34d03625965", "name": "471255a0-9ea3-4828-a373-c34d03625965", "status": "Failed", "startTime": "2026-07-20T17:07:06.252833555Z", "endTime": "2026-07-20T17:26:18.713895489Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create HCP cluster np-skip-minor-9srmsm at version 4.22.6 Unexpected error: <fmt.wrapError \| 0xc000b0a460>: failed to create HCP cluster np-skip-minor-9srmsm: failed waiting for cluster="np-skip-minor-9srmsm" in resourcegroup="rg-np-skip-minor-9srmsm-w59bjd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/358f9e05-fb59-4073-962d-6950d83db1e2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/358f9e05-fb59-4073-962d-6950d83db1e2", "name": "358f9e05-fb59-4073-962d-6950d83db1e2", "status": "Failed", "startTime": "2026-07-20T17:51:40.13869337Z", "endTime": "2026-07-20T18:10:49.418079989Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create HCP cluster np-skip-minor-9srmsm at version 4.22.6 Unexpected error: <fmt.wrapError \| 0xc000b0a460>: failed to create HCP cluster np-skip-minor-9srmsm: failed waiting for cluster="np-skip-minor-9srmsm" in resourcegroup="rg-np-skip-minor-9srmsm-w59bjd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/358f9e05-fb59-4073-962d-6950d83db1e2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/358f9e05-fb59-4073-962d-6950d83db1e2", "name": "358f9e05-fb59-4073-962d-6950d83db1e2", "status": "Failed", "startTime": "2026-07-20T17:51:40.13869337Z", "endTime": "2026-07-20T18:10:49.418079989Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:86]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0xc000b0a620>: failed to create HCP cluster ea-list4r2wmh: failed waiting for cluster="ea-list4r2wmh" in resourcegroup="ea-list-shdd4s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7ffef2d7-4143-40c9-856d-4dc5bda3fbcb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7ffef2d7-4143-40c9-856d-4dc5bda3fbcb", "name": "7ffef2d7-4143-40c9-856d-4dc5bda3fbcb", "status": "Failed", "startTime": "2026-07-20T18:36:08.338350325Z", "endTime": "2026-07-20T18:55:21.966255683Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:86]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0xc000b0a620>: failed to create HCP cluster ea-list4r2wmh: failed waiting for cluster="ea-list4r2wmh" in resourcegroup="ea-list-shdd4s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7ffef2d7-4143-40c9-856d-4dc5bda3fbcb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7ffef2d7-4143-40c9-856d-4dc5bda3fbcb", "name": "7ffef2d7-4143-40c9-856d-4dc5bda3fbcb", "status": "Failed", "startTime": "2026-07-20T18:36:08.338350325Z", "endTime": "2026-07-20T18:55:21.966255683Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-4-20-v9qp6p/cluster-candidate-4-20-wmszpp should provision Unexpected error: <fmt.wrapError \| 0xc000a8e7c0>: failed to create HCP cluster cluster-candidate-4-20-wmszpp: failed waiting for cluster="cluster-candidate-4-20-wmszpp" in resourcegroup="rg-candidate-4-20-v9qp6p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fedb83c8-2aff-49ac-bdf8-eac1959cf8cc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fedb83c8-2aff-49ac-bdf8-eac1959cf8cc", "name": "fedb83c8-2aff-49ac-bdf8-eac1959cf8cc", "status": "Failed", "startTime": "2026-07-20T17:03:58.653081502Z", "endTime": "2026-07-20T17:22:58.501527449Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-4-20-v9qp6p/cluster-candidate-4-20-wmszpp should provision Unexpected error: <fmt.wrapError \| 0xc000a8e7c0>: failed to create HCP cluster cluster-candidate-4-20-wmszpp: failed waiting for cluster="cluster-candidate-4-20-wmszpp" in resourcegroup="rg-candidate-4-20-v9qp6p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fedb83c8-2aff-49ac-bdf8-eac1959cf8cc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fedb83c8-2aff-49ac-bdf8-eac1959cf8cc", "name": "fedb83c8-2aff-49ac-bdf8-eac1959cf8cc", "status": "Failed", "startTime": "2026-07-20T17:03:58.653081502Z", "endTime": "2026-07-20T17:22:58.501527449Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-4-22-tm88nc/cluster-candidate-4-22-4hz5js should provision Unexpected error: <fmt.wrapError \| 0xc000894600>: failed to create HCP cluster cluster-candidate-4-22-4hz5js: failed waiting for cluster="cluster-candidate-4-22-4hz5js" in resourcegroup="rg-candidate-4-22-tm88nc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/71bedfd7-d5d5-41f3-9984-7a70f8375e7e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/71bedfd7-d5d5-41f3-9984-7a70f8375e7e", "name": "71bedfd7-d5d5-41f3-9984-7a70f8375e7e", "status": "Failed", "startTime": "2026-07-20T17:04:54.732277444Z", "endTime": "2026-07-20T17:23:58.453899678Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-4-22-tm88nc/cluster-candidate-4-22-4hz5js should provision Unexpected error: <fmt.wrapError \| 0xc000894600>: failed to create HCP cluster cluster-candidate-4-22-4hz5js: failed waiting for cluster="cluster-candidate-4-22-4hz5js" in resourcegroup="rg-candidate-4-22-tm88nc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/71bedfd7-d5d5-41f3-9984-7a70f8375e7e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/71bedfd7-d5d5-41f3-9984-7a70f8375e7e", "name": "71bedfd7-d5d5-41f3-9984-7a70f8375e7e", "status": "Failed", "startTime": "2026-07-20T17:04:54.732277444Z", "endTime": "2026-07-20T17:23:58.453899678Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:111]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0xc0010083a0>: failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-hm2647" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0b195ba0-db76-482d-82fa-7f8b278c5db4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0b195ba0-db76-482d-82fa-7f8b278c5db4", "name": "0b195ba0-db76-482d-82fa-7f8b278c5db4", "status": "Failed", "startTime": "2026-07-20T17:50:23.335917328Z", "endTime": "2026-07-20T18:09:39.840081605Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:111]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0xc0010083a0>: failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-hm2647" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0b195ba0-db76-482d-82fa-7f8b278c5db4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0b195ba0-db76-482d-82fa-7f8b278c5db4", "name": "0b195ba0-db76-482d-82fa-7f8b278c5db4", "status": "Failed", "startTime": "2026-07-20T17:50:23.335917328Z", "endTime": "2026-07-20T18:09:39.840081605Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test FIPS Mode Support with ExperimentalReleaseFeatures AFEC registered should create an HCP cluster with FIPS mode enabled via experimental tag · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:84]: failed to create HCP cluster "fips-enabled-cluster" with FIPS enabled Unexpected error: <fmt.wrapError \| 0xc000a404e0>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-779bpb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c42f1b6e-7cf0-4e79-9eee-8459407f4c5a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c42f1b6e-7cf0-4e79-9eee-8459407f4c5a", "name": "c42f1b6e-7cf0-4e79-9eee-8459407f4c5a", "status": "Failed", "startTime": "2026-07-20T17:54:15.703797368Z", "endTime": "2026-07-20T18:13:19.446706509Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:84]: failed to create HCP cluster "fips-enabled-cluster" with FIPS enabled Unexpected error: <fmt.wrapError \| 0xc000a404e0>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-779bpb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c42f1b6e-7cf0-4e79-9eee-8459407f4c5a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c42f1b6e-7cf0-4e79-9eee-8459407f4c5a", "name": "c42f1b6e-7cf0-4e79-9eee-8459407f4c5a", "status": "Failed", "startTime": "2026-07-20T17:54:15.703797368Z", "endTime": "2026-07-20T18:13:19.446706509Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:109]: failed to create HCP cluster negative-tests-cluster Unexpected error: <fmt.wrapError \| 0xc000c921e0>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-j64qhc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/d7ef4fec-5edc-4a34-992d-faa523ad2853 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/d7ef4fec-5edc-4a34-992d-faa523ad2853", "name": "d7ef4fec-5edc-4a34-992d-faa523ad2853", "status": "Failed", "startTime": "2026-07-20T17:53:07.63945845Z", "endTime": "2026-07-20T18:12:19.434995424Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:109]: failed to create HCP cluster negative-tests-cluster Unexpected error: <fmt.wrapError \| 0xc000c921e0>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-j64qhc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/d7ef4fec-5edc-4a34-992d-faa523ad2853 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/d7ef4fec-5edc-4a34-992d-faa523ad2853", "name": "d7ef4fec-5edc-4a34-992d-faa523ad2853", "status": "Failed", "startTime": "2026-07-20T17:53:07.63945845Z", "endTime": "2026-07-20T18:12:19.434995424Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:289]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0xc0009fa5e0>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-httjb4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7b2f907a-b2fe-4229-8633-6fc353a1f177 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7b2f907a-b2fe-4229-8633-6fc353a1f177", "name": "7b2f907a-b2fe-4229-8633-6fc353a1f177", "status": "Failed", "startTime": "2026-07-20T17:50:17.547241464Z", "endTime": "2026-07-20T18:09:19.686173299Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:289]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0xc0009fa5e0>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-httjb4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7b2f907a-b2fe-4229-8633-6fc353a1f177 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7b2f907a-b2fe-4229-8633-6fc353a1f177", "name": "7b2f907a-b2fe-4229-8633-6fc353a1f177", "status": "Failed", "startTime": "2026-07-20T17:50:17.547241464Z", "endTime": "2026-07-20T18:09:19.686173299Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:132]: failed to create HCP cluster np-version-upgrade-cluster-gvcrtt with version 4.21.25 Unexpected error: <fmt.wrapError \| 0xc000e544c0>: failed to create HCP cluster np-version-upgrade-cluster-gvcrtt: failed waiting for cluster="np-version-upgrade-cluster-gvcrtt" in resourcegroup="rg-np-version-upgrade-gvcrtt-xqwc5w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5a5a2833-0ca0-4b5d-b11a-4b90da619a8c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5a5a2833-0ca0-4b5d-b11a-4b90da619a8c", "name": "5a5a2833-0ca0-4b5d-b11a-4b90da619a8c", "status": "Failed", "startTime": "2026-07-20T17:51:08.728865403Z", "endTime": "2026-07-20T18:10:19.820799544Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:132]: failed to create HCP cluster np-version-upgrade-cluster-gvcrtt with version 4.21.25 Unexpected error: <fmt.wrapError \| 0xc000e544c0>: failed to create HCP cluster np-version-upgrade-cluster-gvcrtt: failed waiting for cluster="np-version-upgrade-cluster-gvcrtt" in resourcegroup="rg-np-version-upgrade-gvcrtt-xqwc5w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5a5a2833-0ca0-4b5d-b11a-4b90da619a8c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5a5a2833-0ca0-4b5d-b11a-4b90da619a8c", "name": "5a5a2833-0ca0-4b5d-b11a-4b90da619a8c", "status": "Failed", "startTime": "2026-07-20T17:51:08.728865403Z", "endTime": "2026-07-20T18:10:19.820799544Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling Unexpected error: <fmt.wrapError \| 0xc0014ec660>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-6kbj5v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8ad591a9-9e97-433f-b703-aec2e06c3851 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8ad591a9-9e97-433f-b703-aec2e06c3851", "name": "8ad591a9-9e97-433f-b703-aec2e06c3851", "status": "Failed", "startTime": "2026-07-20T16:17:55.004468099Z", "endTime": "2026-07-20T16:36:57.134340873Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling Unexpected error: <fmt.wrapError \| 0xc0014ec660>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-6kbj5v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8ad591a9-9e97-433f-b703-aec2e06c3851 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8ad591a9-9e97-433f-b703-aec2e06c3851", "name": "8ad591a9-9e97-433f-b703-aec2e06c3851", "status": "Failed", "startTime": "2026-07-20T16:17:55.004468099Z", "endTime": "2026-07-20T16:36:57.134340873Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 16:02:17 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: failed to create HCP cluster version 4.19 Unexpected error: <fmt.wrapErrors \| 0xc0016028d0>: failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-9zsl7m" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-9zsl7m, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: failed to create HCP cluster version 4.19 Unexpected error: <fmt.wrapErrors \| 0xc0016028d0>: failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-9zsl7m" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-9zsl7m, error: context deadline exceeded ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:589]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000dee400>: failed to create HCP cluster np-downgrade-zlnpvz: failed waiting for cluster="np-downgrade-zlnpvz" in resourcegroup="rg-np-downgrade-zlnpvz-474nfv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/72b18c43-9bb3-4395-893b-51c33b79054c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/72b18c43-9bb3-4395-893b-51c33b79054c", "name": "72b18c43-9bb3-4395-893b-51c33b79054c", "status": "Failed", "startTime": "2026-07-20T17:51:45.683730612Z", "endTime": "2026-07-20T18:10:49.792207288Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:589]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000dee400>: failed to create HCP cluster np-downgrade-zlnpvz: failed waiting for cluster="np-downgrade-zlnpvz" in resourcegroup="rg-np-downgrade-zlnpvz-474nfv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/72b18c43-9bb3-4395-893b-51c33b79054c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/72b18c43-9bb3-4395-893b-51c33b79054c", "name": "72b18c43-9bb3-4395-893b-51c33b79054c", "status": "Failed", "startTime": "2026-07-20T17:51:45.683730612Z", "endTime": "2026-07-20T18:10:49.792207288Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test Unexpected error: <fmt.wrapError \| 0xc00060b060>: failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-89796h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6f2f4d55-a072-460d-890f-246667225587 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6f2f4d55-a072-460d-890f-246667225587", "name": "6f2f4d55-a072-460d-890f-246667225587", "status": "Failed", "startTime": "2026-07-20T16:18:07.556138017Z", "endTime": "2026-07-20T16:37:17.037192315Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test Unexpected error: <fmt.wrapError \| 0xc00060b060>: failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-89796h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6f2f4d55-a072-460d-890f-246667225587 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6f2f4d55-a072-460d-890f-246667225587", "name": "6f2f4d55-a072-460d-890f-246667225587", "status": "Failed", "startTime": "2026-07-20T16:18:07.556138017Z", "endTime": "2026-07-20T16:37:17.037192315Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:216]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0xc000958680>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-zz6s2n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/cee811fc-ac9c-4c6a-9336-c06a343317b5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/cee811fc-ac9c-4c6a-9336-c06a343317b5", "name": "cee811fc-ac9c-4c6a-9336-c06a343317b5", "status": "Failed", "startTime": "2026-07-20T17:51:53.535071338Z", "endTime": "2026-07-20T18:11:00.258308483Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:216]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0xc000958680>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-zz6s2n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/cee811fc-ac9c-4c6a-9336-c06a343317b5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/cee811fc-ac9c-4c6a-9336-c06a343317b5", "name": "cee811fc-ac9c-4c6a-9336-c06a343317b5", "status": "Failed", "startTime": "2026-07-20T17:51:53.535071338Z", "endTime": "2026-07-20T18:11:00.258308483Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: failed to create HCP cluster "cluster-zstream-4-22-m7pf22" with version 4.22.4 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000e8a240>: failed to create HCP cluster cluster-zstream-4-22-m7pf22: failed waiting for cluster="cluster-zstream-4-22-m7pf22" in resourcegroup="rg-zstream-upgrade-4-22-5754g9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/134663f3-b85b-4f95-aaa5-036030778a08 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/134663f3-b85b-4f95-aaa5-036030778a08", "name": "134663f3-b85b-4f95-aaa5-036030778a08", "status": "Failed", "startTime": "2026-07-20T17:05:35.316811643Z", "endTime": "2026-07-20T17:24:38.212528117Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: failed to create HCP cluster "cluster-zstream-4-22-m7pf22" with version 4.22.4 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000e8a240>: failed to create HCP cluster cluster-zstream-4-22-m7pf22: failed waiting for cluster="cluster-zstream-4-22-m7pf22" in resourcegroup="rg-zstream-upgrade-4-22-5754g9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/134663f3-b85b-4f95-aaa5-036030778a08 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/134663f3-b85b-4f95-aaa5-036030778a08", "name": "134663f3-b85b-4f95-aaa5-036030778a08", "status": "Failed", "startTime": "2026-07-20T17:05:35.316811643Z", "endTime": "2026-07-20T17:24:38.212528117Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:101]: failed to create HCP cluster np-autoscale-cluster Unexpected error: <fmt.wrapError \| 0xc000e14b80>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-9skv5p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/201959ef-a16a-47de-b332-8879a3c4907b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/201959ef-a16a-47de-b332-8879a3c4907b", "name": "201959ef-a16a-47de-b332-8879a3c4907b", "status": "Failed", "startTime": "2026-07-20T17:49:47.640842257Z", "endTime": "2026-07-20T18:08:49.417214158Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:101]: failed to create HCP cluster np-autoscale-cluster Unexpected error: <fmt.wrapError \| 0xc000e14b80>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-9skv5p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/201959ef-a16a-47de-b332-8879a3c4907b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/201959ef-a16a-47de-b332-8879a3c4907b", "name": "201959ef-a16a-47de-b332-8879a3c4907b", "status": "Failed", "startTime": "2026-07-20T17:49:47.640842257Z", "endTime": "2026-07-20T18:08:49.417214158Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: failed to create HCP cluster "cluster-zstream-4-19-45h4f8" with version 4.19.37 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000d76b00>: failed to create HCP cluster cluster-zstream-4-19-45h4f8: failed waiting for cluster="cluster-zstream-4-19-45h4f8" in resourcegroup="rg-zstream-upgrade-4-19-dpg7px" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/98c43fdd-5cd7-4cc7-a362-b77fd854b20a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/98c43fdd-5cd7-4cc7-a362-b77fd854b20a", "name": "98c43fdd-5cd7-4cc7-a362-b77fd854b20a", "status": "Failed", "startTime": "2026-07-20T17:04:53.582585323Z", "endTime": "2026-07-20T17:23:58.453659574Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: failed to create HCP cluster "cluster-zstream-4-19-45h4f8" with version 4.19.37 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000d76b00>: failed to create HCP cluster cluster-zstream-4-19-45h4f8: failed waiting for cluster="cluster-zstream-4-19-45h4f8" in resourcegroup="rg-zstream-upgrade-4-19-dpg7px" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/98c43fdd-5cd7-4cc7-a362-b77fd854b20a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/98c43fdd-5cd7-4cc7-a362-b77fd854b20a", "name": "98c43fdd-5cd7-4cc7-a362-b77fd854b20a", "status": "Failed", "startTime": "2026-07-20T17:04:53.582585323Z", "endTime": "2026-07-20T17:23:58.453659574Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:104]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc000c4ca20>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-ksvhl5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8f09b243-ae51-46ba-959a-9bdddedd1d4d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8f09b243-ae51-46ba-959a-9bdddedd1d4d", "name": "8f09b243-ae51-46ba-959a-9bdddedd1d4d", "status": "Failed", "startTime": "2026-07-20T17:06:30.075955701Z", "endTime": "2026-07-20T17:25:48.195692162Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:104]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc000c4ca20>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-ksvhl5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8f09b243-ae51-46ba-959a-9bdddedd1d4d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8f09b243-ae51-46ba-959a-9bdddedd1d4d", "name": "8f09b243-ae51-46ba-959a-9bdddedd1d4d", "status": "Failed", "startTime": "2026-07-20T17:06:30.075955701Z", "endTime": "2026-07-20T17:25:48.195692162Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:181]: failed to create HCP cluster "cp-ystream-upgrade-4-21-gxlkkp" with install version 4.20.30 Unexpected error: <fmt.wrapError \| 0xc0010e6160>: failed to create HCP cluster cp-ystream-upgrade-4-21-gxlkkp: failed waiting for cluster="cp-ystream-upgrade-4-21-gxlkkp" in resourcegroup="rg-cp-ystream-upgrade-4-21-f4jwjk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/197415b6-02bb-4107-b9cc-08c504e38f8d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/197415b6-02bb-4107-b9cc-08c504e38f8d", "name": "197415b6-02bb-4107-b9cc-08c504e38f8d", "status": "Failed", "startTime": "2026-07-20T17:05:31.092655458Z", "endTime": "2026-07-20T17:24:38.199264603Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:181]: failed to create HCP cluster "cp-ystream-upgrade-4-21-gxlkkp" with install version 4.20.30 Unexpected error: <fmt.wrapError \| 0xc0010e6160>: failed to create HCP cluster cp-ystream-upgrade-4-21-gxlkkp: failed waiting for cluster="cp-ystream-upgrade-4-21-gxlkkp" in resourcegroup="rg-cp-ystream-upgrade-4-21-f4jwjk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/197415b6-02bb-4107-b9cc-08c504e38f8d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/197415b6-02bb-4107-b9cc-08c504e38f8d", "name": "197415b6-02bb-4107-b9cc-08c504e38f8d", "status": "Failed", "startTime": "2026-07-20T17:05:31.092655458Z", "endTime": "2026-07-20T17:24:38.199264603Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: failed to create HCP cluster "cluster-zstream-4-21-6zmtbg" with version 4.21.24 on candidate channel Unexpected error: <fmt.wrapError \| 0xc0009b0900>: failed to create HCP cluster cluster-zstream-4-21-6zmtbg: failed waiting for cluster="cluster-zstream-4-21-6zmtbg" in resourcegroup="rg-zstream-upgrade-4-21-47m5v8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6f1495ef-2de3-43f9-b92e-f0627f8c87ba -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6f1495ef-2de3-43f9-b92e-f0627f8c87ba", "name": "6f1495ef-2de3-43f9-b92e-f0627f8c87ba", "status": "Failed", "startTime": "2026-07-20T17:04:43.154308183Z", "endTime": "2026-07-20T17:23:58.31711227Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: failed to create HCP cluster "cluster-zstream-4-21-6zmtbg" with version 4.21.24 on candidate channel Unexpected error: <fmt.wrapError \| 0xc0009b0900>: failed to create HCP cluster cluster-zstream-4-21-6zmtbg: failed waiting for cluster="cluster-zstream-4-21-6zmtbg" in resourcegroup="rg-zstream-upgrade-4-21-47m5v8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6f1495ef-2de3-43f9-b92e-f0627f8c87ba -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6f1495ef-2de3-43f9-b92e-f0627f8c87ba", "name": "6f1495ef-2de3-43f9-b92e-f0627f8c87ba", "status": "Failed", "startTime": "2026-07-20T17:04:43.154308183Z", "endTime": "2026-07-20T17:23:58.31711227Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred Deserialization Error: no output from command 16:02:17 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite prod/parallel Show raw failure 3089fb7 io.ReadAtLeast({0x92938e0, 0xc0014ab680}, {0xc0014b4204, 0x9, 0x9}, 0x9) /usr/lib/golang/src/io/io.go:335 +0x8e fp=0xc000af7d00 sp=0xc000af7cb8 pc=0x2f7e06e io.ReadFull(...) /usr/lib/golang/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0xc0014b4204, 0x9, 0xc000000063?}, {0x92938e0?, 0xc0014ab680?}) /go/pkg/mod/golang.org/x/net@v0.55.0/http2/frame.go:250 +0x65 fp=0xc000af7d50 sp=0xc000af7d00 pc=0x35ee225 golang.org/x/net/http2.(Framer).ReadFrameHeader(0xc0014b41c0) /go/pkg/mod/golang.org/x/net@v0.55.0/http2/frame.go:513 +0x6b fp=0xc000af7dd8 sp=0xc000af7d50 pc=0x35eedeb golang.org/x/net/http2.(Framer).ReadFrame(0xc0014b41c0) /go/pkg/mod/golang.org/x/net@v0.55.0/http2/frame.go:572 +0x18 fp=0xc000af7e18 sp=0xc000af7dd8 pc=0x35ef558 golang.org/x/net/http2.(clientConnReadLoop).run(0xc000af7fa8) /go/pkg/mod/golang.org/x/net@v0.55.0/http2/transport.go:1878 +0xca fp=0xc000af7f60 sp=0xc000af7e18 pc=0x360154a golang.org/x/net/http2.(ClientConn).readLoop(0xc0016c0a80) /go/pkg/mod/golang.org/x/net@v0.55.0/http2/transport.go:1760 +0x52 fp=0xc000af7fc8 sp=0xc000af7f60 pc=0x3600a72 golang.org/x/net/http2.(Transport).newClientConn.gowrap1() /go/pkg/mod/golang.org/x/net@v0.55.0/http2/transport.go:554 +0x25 fp=0xc000af7fe0 sp=0xc000af7fc8 pc=0x35fa1a5 runtime.goexit({}) /usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000af7fe8 sp=0xc000af7fe0 pc=0x2f2bcc1 created by golang.org/x/net/http2.(Transport).newClientConn in goroutine 134 /go/pkg/mod/golang.org/x/net@v0.55.0/http2/transport.go:554 +0xda5 goroutine 316 gp=0xc000697dc0 m=nil [sync.WaitGroup.Wait]: runtime.gopark(0xc0016115b0?, 0x2eba874?, 0x20?, 0x99?, 0x200c001546540?) /usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc001611550 sp=0xc001611530 pc=0x2f2350e runtime.goparkunlock(...) /usr/lib/golang/src/runtime/proc.go:466 runtime.semacquire1(0xc001558348, 0x0, 0x1, 0x0, 0x19) /usr/lib/golang/src/runtime/sema.go:192 +0x21d fp=0xc0016115b8 sp=0xc001611550 pc=0x2eff59d sync.runtime_SemacquireWaitGroup(0xc000d25920?, 0xc0?) /usr/lib/golang/src/runtime/sema.go:114 +0x2e fp=0xc0016115f0 sp=0xc0016115b8 pc=0x2f24c2e sync.(WaitGroup).Wait(0xc001558340) /usr/lib/golang/src/sync/waitgroup.go:206 +0x85 fp=0xc001611618 sp=0xc0016115f0 pc=0x2f48405 github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).CleanupResourceGroups(0xc000eba000, {0x7f6c39063a60, 0xc0015a32c0}, {{0xc000ec09c0, 0x1, 0x1}, 0x34630b8a000, {0x871a24d, 0x8}, {{0x0, ...}, ...}}) /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:371 +0x252 fp=0xc001611730 sp=0xc001611618 pc=0x6c9d292 github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).deleteCreatedResources(0xc000eba000, {0x7f6c39063a60, 0xc0015a32c0}) /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:262 +0x250 fp=0xc001611888 sp=0xc001611730 pc=0x6c9c9f0 github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).deleteCreatedResources-fm({0x7f6c39063a60?, 0xc0015a32c0?}) <autogenerated>:1 +0x33 fp=0xc0016118b0 sp=0xc001611888 pc=0x6cbcad3 runtime.call16(0xc000e94300, 0xc000ae0410, 0x0, 0x0, 0x0, 0x10, 0xc001611b08) /usr/lib/golang/src/runtime/asm_amd64.s:774 +0x43 fp=0xc0016118d0 sp=0xc0016118b0 pc=0x2f2a223 runtime.reflectcall(0x80e5200?, 0xc0015a32c0?, 0x2?, 0x8751e1d?, 0x0?, 0x12?, 0x7c9d5e0?) <autogenerated>:1 +0x36 fp=0xc001611910 sp=0xc0016118d0 pc=0x2f32ab6 reflect.Value.call({0x75465a0?, 0xc000ae0410?, 0xc000e8c780?}, {0x870dbd9, 0x4}, {0xc000879f60, 0x1, 0xc001476a00?}) /usr/lib/golang/src/reflect/value.go:581 +0xcc6 fp=0xc001611e70 sp=0xc001611910 pc=0x2f65c66 reflect.Value.Call({0x75465a0?, 0xc000ae0410?, 0xc000879f68?}, {0xc000879f60?, 0xc000eba000?, 0x932c968?}) /usr/lib/golang/src/reflect/value.go:365 +0xb9 fp=0xc001611ee8 sp=0xc001611e70 pc=0x2f64d59 github.com/onsi/ginkgo/v2/internal.NewCleanupNode.func2({0x93472c8?, 0xc0015a32c0?}) /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:589 +0x150 fp=0xc001611f88 sp=0xc001611ee8 pc=0x3388c70 github.com/onsi/ginkgo/v2/internal.(Suite).runNode.func3() /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc001611fe0 sp=0xc001611f88 pc=0x339b73b runtime.goexit({}) /usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001611fe8 sp=0xc001611fe0 pc=0x2f2bcc1 created by github.com/onsi/ginkgo/v2/internal.(Suite).runNode in goroutine 83 /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50 rax 0xca rbx 0x0 rcx 0x2f2dae3 rdx 0x0 rdi 0xc9599f0 rsi 0x80 rbp 0x7ffe78eb87a8 rsp 0x7ffe78eb8760 r8 0x0 r9 0x0 r10 0x0 r11 0x286 r12 0x4c r13 0x1 r14 0xc957640 r15 0x1 rip 0x2f2dae1 rflags 0x286 cs 0x33 fs 0x0 gs 0x0 Command Error: exit status 2 Deserialization Error: no output from command 3089fb7 io.ReadAtLeast({0x92938e0, 0xc0014ab680}, {0xc0014b4204, 0x9, 0x9}, 0x9) /usr/lib/golang/src/io/io.go:335 +0x8e fp=0xc000af7d00 sp=0xc000af7cb8 pc=0x2f7e06e io.ReadFull(...) /usr/lib/golang/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0xc0014b4204, 0x9, 0xc000000063?}, {0x92938e0?, 0xc0014ab680?}) /go/pkg/mod/golang.org/x/net@v0.55.0/http2/frame.go:250 +0x65 fp=0xc000af7d50 sp=0xc000af7d00 pc=0x35ee225 golang.org/x/net/http2.(Framer).ReadFrameHeader(0xc0014b41c0) /go/pkg/mod/golang.org/x/net@v0.55.0/http2/frame.go:513 +0x6b fp=0xc000af7dd8 sp=0xc000af7d50 pc=0x35eedeb golang.org/x/net/http2.(Framer).ReadFrame(0xc0014b41c0) /go/pkg/mod/golang.org/x/net@v0.55.0/http2/frame.go:572 +0x18 fp=0xc000af7e18 sp=0xc000af7dd8 pc=0x35ef558 golang.org/x/net/http2.(clientConnReadLoop).run(0xc000af7fa8) /go/pkg/mod/golang.org/x/net@v0.55.0/http2/transport.go:1878 +0xca fp=0xc000af7f60 sp=0xc000af7e18 pc=0x360154a golang.org/x/net/http2.(ClientConn).readLoop(0xc0016c0a80) /go/pkg/mod/golang.org/x/net@v0.55.0/http2/transport.go:1760 +0x52 fp=0xc000af7fc8 sp=0xc000af7f60 pc=0x3600a72 golang.org/x/net/http2.(Transport).newClientConn.gowrap1() /go/pkg/mod/golang.org/x/net@v0.55.0/http2/transport.go:554 +0x25 fp=0xc000af7fe0 sp=0xc000af7fc8 pc=0x35fa1a5 runtime.goexit({}) /usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000af7fe8 sp=0xc000af7fe0 pc=0x2f2bcc1 created by golang.org/x/net/http2.(Transport).newClientConn in goroutine 134 /go/pkg/mod/golang.org/x/net@v0.55.0/http2/transport.go:554 +0xda5 goroutine 316 gp=0xc000697dc0 m=nil [sync.WaitGroup.Wait]: runtime.gopark(0xc0016115b0?, 0x2eba874?, 0x20?, 0x99?, 0x200c001546540?) /usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc001611550 sp=0xc001611530 pc=0x2f2350e runtime.goparkunlock(...) /usr/lib/golang/src/runtime/proc.go:466 runtime.semacquire1(0xc001558348, 0x0, 0x1, 0x0, 0x19) /usr/lib/golang/src/runtime/sema.go:192 +0x21d fp=0xc0016115b8 sp=0xc001611550 pc=0x2eff59d sync.runtime_SemacquireWaitGroup(0xc000d25920?, 0xc0?) /usr/lib/golang/src/runtime/sema.go:114 +0x2e fp=0xc0016115f0 sp=0xc0016115b8 pc=0x2f24c2e sync.(WaitGroup).Wait(0xc001558340) /usr/lib/golang/src/sync/waitgroup.go:206 +0x85 fp=0xc001611618 sp=0xc0016115f0 pc=0x2f48405 github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).CleanupResourceGroups(0xc000eba000, {0x7f6c39063a60, 0xc0015a32c0}, {{0xc000ec09c0, 0x1, 0x1}, 0x34630b8a000, {0x871a24d, 0x8}, {{0x0, ...}, ...}}) /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:371 +0x252 fp=0xc001611730 sp=0xc001611618 pc=0x6c9d292 github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).deleteCreatedResources(0xc000eba000, {0x7f6c39063a60, 0xc0015a32c0}) /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:262 +0x250 fp=0xc001611888 sp=0xc001611730 pc=0x6c9c9f0 github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).deleteCreatedResources-fm({0x7f6c39063a60?, 0xc0015a32c0?}) <autogenerated>:1 +0x33 fp=0xc0016118b0 sp=0xc001611888 pc=0x6cbcad3 runtime.call16(0xc000e94300, 0xc000ae0410, 0x0, 0x0, 0x0, 0x10, 0xc001611b08) /usr/lib/golang/src/runtime/asm_amd64.s:774 +0x43 fp=0xc0016118d0 sp=0xc0016118b0 pc=0x2f2a223 runtime.reflectcall(0x80e5200?, 0xc0015a32c0?, 0x2?, 0x8751e1d?, 0x0?, 0x12?, 0x7c9d5e0?) <autogenerated>:1 +0x36 fp=0xc001611910 sp=0xc0016118d0 pc=0x2f32ab6 reflect.Value.call({0x75465a0?, 0xc000ae0410?, 0xc000e8c780?}, {0x870dbd9, 0x4}, {0xc000879f60, 0x1, 0xc001476a00?}) /usr/lib/golang/src/reflect/value.go:581 +0xcc6 fp=0xc001611e70 sp=0xc001611910 pc=0x2f65c66 reflect.Value.Call({0x75465a0?, 0xc000ae0410?, 0xc000879f68?}, {0xc000879f60?, 0xc000eba000?, 0x932c968?}) /usr/lib/golang/src/reflect/value.go:365 +0xb9 fp=0xc001611ee8 sp=0xc001611e70 pc=0x2f64d59 github.com/onsi/ginkgo/v2/internal.NewCleanupNode.func2({0x93472c8?, 0xc0015a32c0?}) /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:589 +0x150 fp=0xc001611f88 sp=0xc001611ee8 pc=0x3388c70 github.com/onsi/ginkgo/v2/internal.(Suite).runNode.func3() /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc001611fe0 sp=0xc001611f88 pc=0x339b73b runtime.goexit({}) /usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001611fe8 sp=0xc001611fe0 pc=0x2f2bcc1 created by github.com/onsi/ginkgo/v2/internal.(Suite).runNode in goroutine 83 /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50 rax 0xca rbx 0x0 rcx 0x2f2dae3 rdx 0x0 rdi 0xc9599f0 rsi 0x80 rbp 0x7ffe78eb87a8 rsp 0x7ffe78eb8760 r8 0x0 r9 0x0 r10 0x0 r11 0x286 r12 0x4c r13 0x1 r14 0xc957640 r15 0x1 rip 0x2f2dae1 rflags 0x286 cs 0x33 fs 0x0 gs 0x0 Command Error: exit status 2 Deserialization Error: no output from command ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91]: failed to create HCP cluster cluster-np-labels-taints Unexpected error: <fmt.wrapError \| 0xc000c0ae40>: failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-lffq4b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e551f69f-3ad8-4324-975f-6a196a46c1de -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e551f69f-3ad8-4324-975f-6a196a46c1de", "name": "e551f69f-3ad8-4324-975f-6a196a46c1de", "status": "Failed", "startTime": "2026-07-20T17:51:00.807585389Z", "endTime": "2026-07-20T18:10:09.420000126Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91]: failed to create HCP cluster cluster-np-labels-taints Unexpected error: <fmt.wrapError \| 0xc000c0ae40>: failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-lffq4b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e551f69f-3ad8-4324-975f-6a196a46c1de -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e551f69f-3ad8-4324-975f-6a196a46c1de", "name": "e551f69f-3ad8-4324-975f-6a196a46c1de", "status": "Failed", "startTime": "2026-07-20T17:51:00.807585389Z", "endTime": "2026-07-20T18:10:09.420000126Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred Cluster provisioning failed 16:02:17 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:157]: Cluster provisioning failed fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:157]: Cluster provisioning failed ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-5-0-54sh25/cluster-candidate-5-0-2qxdcj should provision Unexpected error: <fmt.wrapError \| 0xc000a64f80>: failed to create HCP cluster cluster-candidate-5-0-2qxdcj: failed waiting for cluster="cluster-candidate-5-0-2qxdcj" in resourcegroup="rg-candidate-5-0-54sh25" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e3fc52e2-6273-4c6b-b9fe-958a375e9be2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e3fc52e2-6273-4c6b-b9fe-958a375e9be2", "name": "e3fc52e2-6273-4c6b-b9fe-958a375e9be2", "status": "Failed", "startTime": "2026-07-20T17:04:41.800474139Z", "endTime": "2026-07-20T17:23:58.315072737Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-5-0-54sh25/cluster-candidate-5-0-2qxdcj should provision Unexpected error: <fmt.wrapError \| 0xc000a64f80>: failed to create HCP cluster cluster-candidate-5-0-2qxdcj: failed waiting for cluster="cluster-candidate-5-0-2qxdcj" in resourcegroup="rg-candidate-5-0-54sh25" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e3fc52e2-6273-4c6b-b9fe-958a375e9be2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e3fc52e2-6273-4c6b-b9fe-958a375e9be2", "name": "e3fc52e2-6273-4c6b-b9fe-958a375e9be2", "status": "Failed", "startTime": "2026-07-20T17:04:41.800474139Z", "endTime": "2026-07-20T17:23:58.315072737Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:132]: failed to create HCP cluster np-version-upgrade-cluster-2v9hgq with version 4.20.30 Unexpected error: <fmt.wrapError \| 0xc000cc50c0>: failed to create HCP cluster np-version-upgrade-cluster-2v9hgq: failed waiting for cluster="np-version-upgrade-cluster-2v9hgq" in resourcegroup="rg-np-version-upgrade-2v9hgq-mnksn4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/90cd55f5-fa73-4d3c-a716-f849f4605a13 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/90cd55f5-fa73-4d3c-a716-f849f4605a13", "name": "90cd55f5-fa73-4d3c-a716-f849f4605a13", "status": "Failed", "startTime": "2026-07-20T17:50:58.800649836Z", "endTime": "2026-07-20T18:10:00.222985677Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:132]: failed to create HCP cluster np-version-upgrade-cluster-2v9hgq with version 4.20.30 Unexpected error: <fmt.wrapError \| 0xc000cc50c0>: failed to create HCP cluster np-version-upgrade-cluster-2v9hgq: failed waiting for cluster="np-version-upgrade-cluster-2v9hgq" in resourcegroup="rg-np-version-upgrade-2v9hgq-mnksn4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/90cd55f5-fa73-4d3c-a716-f849f4605a13 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/90cd55f5-fa73-4d3c-a716-f849f4605a13", "name": "90cd55f5-fa73-4d3c-a716-f849f4605a13", "status": "Failed", "startTime": "2026-07-20T17:50:58.800649836Z", "endTime": "2026-07-20T18:10:00.222985677Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:269]: failed to create HCP cluster shoebox-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000c32b40>: failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-l6nk7z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/d3134b27-f3d1-47f1-9586-af4dc6c51520 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/d3134b27-f3d1-47f1-9586-af4dc6c51520", "name": "d3134b27-f3d1-47f1-9586-af4dc6c51520", "status": "Failed", "startTime": "2026-07-20T17:52:34.467585628Z", "endTime": "2026-07-20T18:11:39.44183724Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:269]: failed to create HCP cluster shoebox-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000c32b40>: failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-l6nk7z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/d3134b27-f3d1-47f1-9586-af4dc6c51520 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/d3134b27-f3d1-47f1-9586-af4dc6c51520", "name": "d3134b27-f3d1-47f1-9586-af4dc6c51520", "status": "Failed", "startTime": "2026-07-20T17:52:34.467585628Z", "endTime": "2026-07-20T18:11:39.44183724Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift 16:02:17 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:150]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs Unexpected error: <fmt.wrapError \| 0xc000ced480>: failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-pll2br" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8d2bc806-e523-4d6f-b758-feebf4bbe60f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8d2bc806-e523-4d6f-b758-feebf4bbe60f", "name": "8d2bc806-e523-4d6f-b758-feebf4bbe60f", "status": "Failed", "startTime": "2026-07-20T16:19:24.843029182Z", "endTime": "2026-07-20T16:38:26.889267813Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:150]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs Unexpected error: <fmt.wrapError \| 0xc000ced480>: failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-pll2br" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8d2bc806-e523-4d6f-b758-feebf4bbe60f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8d2bc806-e523-4d6f-b758-feebf4bbe60f", "name": "8d2bc806-e523-4d6f-b758-feebf4bbe60f", "status": "Failed", "startTime": "2026-07-20T16:19:24.843029182Z", "endTime": "2026-07-20T16:38:26.889267813Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content" } } -------------------------------------------------------------------------------- ... occurred
12:23:52	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
12:19:48	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
12:18:51	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
12:10:29	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
08:38:32	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
05:24:28	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	1	e2e (1) Cluster provisioning failed 05:24:28 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:157]: Cluster provisioning failed fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:157]: Cluster provisioning failed
02:00:52	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	2	e2e (2) DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded 02:00:52 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:283]: Unexpected error: <errors.joinError \| 0xc000a9c570>: failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-candidate-5-0-zkvjxb", caused by: failed waiting for hcpCluster="cluster-candidate-5-0-bh8jcp" in resourcegroup="rg-candidate-5-0-zkvjxb" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-5-0-bh8jcp in resource group rg-candidate-5-0-zkvjxb, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-candidate-5-0-bh8jcp" in resourcegroup="rg-candidate-5-0-zkvjxb" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-5-0-bh8jcp in resource group rg-candidate-5-0-zkvjxb, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:283]: Unexpected error: <errors.joinError \| 0xc000a9c570>: failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-candidate-5-0-zkvjxb", caused by: failed waiting for hcpCluster="cluster-candidate-5-0-bh8jcp" in resourcegroup="rg-candidate-5-0-zkvjxb" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-5-0-bh8jcp in resource group rg-candidate-5-0-zkvjxb, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-candidate-5-0-bh8jcp" in resourcegroup="rg-candidate-5-0-zkvjxb" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-5-0-bh8jcp in resource group rg-candidate-5-0-zkvjxb, error: context deadline exceeded ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 02:00:52 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster" Unexpected error: <errors.joinError \| 0xc0012881e0>: VerifyNodesReady failed: no nodes found in the cluster VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-8d968x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3888477a-bc90-496f-be5a-132321ab3a6f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3888477a-bc90-496f-be5a-132321ab3a6f", "name": "3888477a-bc90-496f-be5a-132321ab3a6f", "status": "Failed", "startTime": "2026-07-20T02:16:49.779634198Z", "endTime": "2026-07-20T02:35:49.961113313Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster" Unexpected error: <errors.joinError \| 0xc0012881e0>: VerifyNodesReady failed: no nodes found in the cluster VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-8d968x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3888477a-bc90-496f-be5a-132321ab3a6f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3888477a-bc90-496f-be5a-132321ab3a6f", "name": "3888477a-bc90-496f-be5a-132321ab3a6f", "status": "Failed", "startTime": "2026-07-20T02:16:49.779634198Z", "endTime": "2026-07-20T02:35:49.961113313Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred