CIHealth
<
Single Day: Aug 19▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: PROD

Time (UTC)JobFailed atResultPRFailed testsDetails
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift
23:35:30 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1
Unexpected error:
    <*fmt.wrapError | 0xc000f5dc80>: 
    failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-sj5szv8t6s8s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/37c1aadb-f927-4c30-95d8-892451317321
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/37c1aadb-f927-4c30-95d8-892451317321",
      "name": "37c1aadb-f927-4c30-95d8-892451317321",
      "status": "Failed",
      "startTime": "2026-08-20T00:43:05.670558184Z",
      "endTime": "2026-08-20T01:02:11.662741691Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1
Unexpected error:
    <*fmt.wrapError | 0xc000f5dc80>: 
    failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-sj5szv8t6s8s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/37c1aadb-f927-4c30-95d8-892451317321
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/37c1aadb-f927-4c30-95d8-892451317321",
      "name": "37c1aadb-f927-4c30-95d8-892451317321",
      "status": "Failed",
      "startTime": "2026-08-20T00:43:05.670558184Z",
      "endTime": "2026-08-20T01:02:11.662741691Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 6
e2e (6)
ERROR CODE: InvalidRequestContent; detail message [controlPlaneDesiredVersionResolution] no upgrade path found from 4.22.0-0.nightly-multi-2026-08-17-010411 to 4.23.0: no reachable versions in target minor and no gateway version in current minor; provider Microsoft.RedHatOpenShift
23:00:07 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:174]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-d5pz9g" to 4.23.0-0.nightly-multi-2026-08-19-140455
Unexpected error:
    <*fmt.wrapError | 0xc000900d00>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-23-d5pz9g" in resourcegroup="rg-cp-ystream-upgrade-4-23-ttgbbn52wj5b" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f80b855b-6ddb-4d41-bcfd-69a629fa5f3a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f80b855b-6ddb-4d41-bcfd-69a629fa5f3a",
      "name": "f80b855b-6ddb-4d41-bcfd-69a629fa5f3a",
      "status": "Failed",
      "startTime": "2026-08-19T23:46:03.25868454Z",
      "endTime": "2026-08-19T23:46:13.75881135Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 4.22.0-0.nightly-multi-2026-08-17-010411 to 4.23.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:174]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-d5pz9g" to 4.23.0-0.nightly-multi-2026-08-19-140455
Unexpected error:
    <*fmt.wrapError | 0xc000900d00>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-23-d5pz9g" in resourcegroup="rg-cp-ystream-upgrade-4-23-ttgbbn52wj5b" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f80b855b-6ddb-4d41-bcfd-69a629fa5f3a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f80b855b-6ddb-4d41-bcfd-69a629fa5f3a",
      "name": "f80b855b-6ddb-4d41-bcfd-69a629fa5f3a",
      "status": "Failed",
      "startTime": "2026-08-19T23:46:03.25868454Z",
      "endTime": "2026-08-19T23:46:13.75881135Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 4.22.0-0.nightly-multi-2026-08-17-010411 to 4.23.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
23:00:07 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:183]: Timed out after 2700.005s.
Expected success, but got an error:
    <*errors.joinError | 0xc001237a58>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-08-18-203845, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:183]: Timed out after 2700.005s.
Expected success, but got an error:
    <*errors.joinError | 0xc001237a58>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-08-18-203845, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
23:00:07 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:183]: Timed out after 2700.000s.
Expected success, but got an error:
    <*errors.joinError | 0xc001a0e480>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.21.0-0.nightly-multi-2026-08-19-155224, targetMinor=4.22.0-0.nightly-multi-2026-08-17-010411) failed: clusterversion status.history has no version in target minor "4.22.0-0.nightly-multi-2026-08-17-010411"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:183]: Timed out after 2700.000s.
Expected success, but got an error:
    <*errors.joinError | 0xc001a0e480>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.21.0-0.nightly-multi-2026-08-19-155224, targetMinor=4.22.0-0.nightly-multi-2026-08-17-010411) failed: clusterversion status.history has no version in target minor "4.22.0-0.nightly-multi-2026-08-17-010411"
    ...
timeout during CreateHCPClusterAndWait; context deadline exceeded
23:00:07 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.1 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-lvzfdk9lcwz6/cluster-nightly-5-1-jcl8vb should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc0017819e0>: 
    failed to create HCP cluster cluster-nightly-5-1-jcl8vb, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-1-jcl8vb in resource group rg-nightly-5-1-lvzfdk9lcwz6, error: failed waiting for cluster="cluster-nightly-5-1-jcl8vb" in resourcegroup="rg-nightly-5-1-lvzfdk9lcwz6" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-1-jcl8vb in resource group rg-nightly-5-1-lvzfdk9lcwz6, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-lvzfdk9lcwz6/cluster-nightly-5-1-jcl8vb should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc0017819e0>: 
    failed to create HCP cluster cluster-nightly-5-1-jcl8vb, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-1-jcl8vb in resource group rg-nightly-5-1-lvzfdk9lcwz6, error: failed waiting for cluster="cluster-nightly-5-1-jcl8vb" in resourcegroup="rg-nightly-5-1-lvzfdk9lcwz6" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-1-jcl8vb in resource group rg-nightly-5-1-lvzfdk9lcwz6, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
23:00:07 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapErrors | 0xc002d08090>: 
    failed to create HCP cluster private-kas, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-g5sqtbgcx787, error: failed waiting for cluster="private-kas" in resourcegroup="private-kas-g5sqtbgcx787" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-g5sqtbgcx787, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapErrors | 0xc002d08090>: 
    failed to create HCP cluster private-kas, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-g5sqtbgcx787, error: failed waiting for cluster="private-kas" in resourcegroup="private-kas-g5sqtbgcx787" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-g5sqtbgcx787, error: context deadline exceeded
    ...
occurred
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
23:00:07 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:183]: Timed out after 2700.006s.
Expected success, but got an error:
    <*errors.joinError | 0xc001651770>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20.0-0.nightly-multi-2026-08-18-064405, targetMinor=4.21.0-0.nightly-multi-2026-08-19-155224) failed: clusterversion status.history has no version in target minor "4.21.0-0.nightly-multi-2026-08-19-155224"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:183]: Timed out after 2700.006s.
Expected success, but got an error:
    <*errors.joinError | 0xc001651770>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20.0-0.nightly-multi-2026-08-18-064405, targetMinor=4.21.0-0.nightly-multi-2026-08-19-155224) failed: clusterversion status.history has no version in target minor "4.21.0-0.nightly-multi-2026-08-19-155224"
    ...
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
x509: certificate signed by unknown authority
10:34:42 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:236]: Timed out after 600.002s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <x509.UnknownAuthorityError>: 
    x509: certificate signed by unknown authority
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:236]: Timed out after 600.002s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <x509.UnknownAuthorityError>: 
    x509: certificate signed by unknown authority
    ...
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 49
e2e (49)
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:593]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f084a0>: 
    failed to create HCP cluster np-downgrade-bb2pdp: failed waiting for cluster="np-downgrade-bb2pdp" in resourcegroup="rg-np-downgrade-bb2pdp-78j5hq9mwzxz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b063fb82-4df1-44ba-b542-bd61ed70bc8d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b063fb82-4df1-44ba-b542-bd61ed70bc8d",
      "name": "b063fb82-4df1-44ba-b542-bd61ed70bc8d",
      "status": "Failed",
      "startTime": "2026-08-19T09:57:38.906376214Z",
      "endTime": "2026-08-19T10:16:40.410448062Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:593]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f084a0>: 
    failed to create HCP cluster np-downgrade-bb2pdp: failed waiting for cluster="np-downgrade-bb2pdp" in resourcegroup="rg-np-downgrade-bb2pdp-78j5hq9mwzxz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b063fb82-4df1-44ba-b542-bd61ed70bc8d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b063fb82-4df1-44ba-b542-bd61ed70bc8d",
      "name": "b063fb82-4df1-44ba-b542-bd61ed70bc8d",
      "status": "Failed",
      "startTime": "2026-08-19T09:57:38.906376214Z",
      "endTime": "2026-08-19T10:16:40.410448062Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc001018e00>: 
    failed to create HCP cluster ea-list25nbjz: failed waiting for cluster="ea-list25nbjz" in resourcegroup="ea-list-v6kgkw2pv2fn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c11e61c-fee1-4307-82fc-61f88ab18015
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c11e61c-fee1-4307-82fc-61f88ab18015",
      "name": "3c11e61c-fee1-4307-82fc-61f88ab18015",
      "status": "Failed",
      "startTime": "2026-08-19T09:14:50.974745851Z",
      "endTime": "2026-08-19T09:33:58.71780003Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc001018e00>: 
    failed to create HCP cluster ea-list25nbjz: failed waiting for cluster="ea-list25nbjz" in resourcegroup="ea-list-v6kgkw2pv2fn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c11e61c-fee1-4307-82fc-61f88ab18015
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c11e61c-fee1-4307-82fc-61f88ab18015",
      "name": "3c11e61c-fee1-4307-82fc-61f88ab18015",
      "status": "Failed",
      "startTime": "2026-08-19T09:14:50.974745851Z",
      "endTime": "2026-08-19T09:33:58.71780003Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-4-21-8l8nzk7rcr74/cluster-candidate-4-21-w5vkj5 should provision
Unexpected error:
    <*fmt.wrapError | 0xc001492600>: 
    failed to create HCP cluster cluster-candidate-4-21-w5vkj5: failed waiting for cluster="cluster-candidate-4-21-w5vkj5" in resourcegroup="rg-candidate-4-21-8l8nzk7rcr74" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/417aa047-dc26-411c-833d-6e166e32eaa3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/417aa047-dc26-411c-833d-6e166e32eaa3",
      "name": "417aa047-dc26-411c-833d-6e166e32eaa3",
      "status": "Failed",
      "startTime": "2026-08-19T09:10:59.38788186Z",
      "endTime": "2026-08-19T09:30:08.769395572Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-4-21-8l8nzk7rcr74/cluster-candidate-4-21-w5vkj5 should provision
Unexpected error:
    <*fmt.wrapError | 0xc001492600>: 
    failed to create HCP cluster cluster-candidate-4-21-w5vkj5: failed waiting for cluster="cluster-candidate-4-21-w5vkj5" in resourcegroup="rg-candidate-4-21-8l8nzk7rcr74" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/417aa047-dc26-411c-833d-6e166e32eaa3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/417aa047-dc26-411c-833d-6e166e32eaa3",
      "name": "417aa047-dc26-411c-833d-6e166e32eaa3",
      "status": "Failed",
      "startTime": "2026-08-19T09:10:59.38788186Z",
      "endTime": "2026-08-19T09:30:08.769395572Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00111aa40>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-rhnvsmn2tmgw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/47dba64a-f0d4-48ab-a8ae-a8b70731d057
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/47dba64a-f0d4-48ab-a8ae-a8b70731d057",
      "name": "47dba64a-f0d4-48ab-a8ae-a8b70731d057",
      "status": "Failed",
      "startTime": "2026-08-19T10:00:42.858469451Z",
      "endTime": "2026-08-19T10:19:50.439535037Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00111aa40>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-rhnvsmn2tmgw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/47dba64a-f0d4-48ab-a8ae-a8b70731d057
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/47dba64a-f0d4-48ab-a8ae-a8b70731d057",
      "name": "47dba64a-f0d4-48ab-a8ae-a8b70731d057",
      "status": "Failed",
      "startTime": "2026-08-19T10:00:42.858469451Z",
      "endTime": "2026-08-19T10:19:50.439535037Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000f92160>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-m56r5mgzh6kq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ece2033d-3c6c-47ee-b731-02562895a746
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ece2033d-3c6c-47ee-b731-02562895a746",
      "name": "ece2033d-3c6c-47ee-b731-02562895a746",
      "status": "Failed",
      "startTime": "2026-08-19T08:24:45.970611075Z",
      "endTime": "2026-08-19T08:43:47.459673973Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000f92160>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-m56r5mgzh6kq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ece2033d-3c6c-47ee-b731-02562895a746
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ece2033d-3c6c-47ee-b731-02562895a746",
      "name": "ece2033d-3c6c-47ee-b731-02562895a746",
      "status": "Failed",
      "startTime": "2026-08-19T08:24:45.970611075Z",
      "endTime": "2026-08-19T08:43:47.459673973Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc00100c480>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-7h6p54bwj22m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15bda795-30c3-473c-85e3-ec66bf5bdac7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15bda795-30c3-473c-85e3-ec66bf5bdac7",
      "name": "15bda795-30c3-473c-85e3-ec66bf5bdac7",
      "status": "Failed",
      "startTime": "2026-08-19T09:58:48.847994826Z",
      "endTime": "2026-08-19T10:17:50.474405815Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc00100c480>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-7h6p54bwj22m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15bda795-30c3-473c-85e3-ec66bf5bdac7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15bda795-30c3-473c-85e3-ec66bf5bdac7",
      "name": "15bda795-30c3-473c-85e3-ec66bf5bdac7",
      "status": "Failed",
      "startTime": "2026-08-19T09:58:48.847994826Z",
      "endTime": "2026-08-19T10:17:50.474405815Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:182]: failed to create HCP cluster "cp-ystream-upgrade-4-22-rp4hdh" with install version 4.21.28
Unexpected error:
    <*fmt.wrapError | 0xc000a91a20>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-rp4hdh: failed waiting for cluster="cp-ystream-upgrade-4-22-rp4hdh" in resourcegroup="rg-cp-ystream-upgrade-4-22-kl85575v2vj5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44736fc3-da7f-4aaf-947d-ebb9c821d228
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44736fc3-da7f-4aaf-947d-ebb9c821d228",
      "name": "44736fc3-da7f-4aaf-947d-ebb9c821d228",
      "status": "Failed",
      "startTime": "2026-08-19T09:12:16.108956001Z",
      "endTime": "2026-08-19T09:31:18.830785986Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:182]: failed to create HCP cluster "cp-ystream-upgrade-4-22-rp4hdh" with install version 4.21.28
Unexpected error:
    <*fmt.wrapError | 0xc000a91a20>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-rp4hdh: failed waiting for cluster="cp-ystream-upgrade-4-22-rp4hdh" in resourcegroup="rg-cp-ystream-upgrade-4-22-kl85575v2vj5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44736fc3-da7f-4aaf-947d-ebb9c821d228
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44736fc3-da7f-4aaf-947d-ebb9c821d228",
      "name": "44736fc3-da7f-4aaf-947d-ebb9c821d228",
      "status": "Failed",
      "startTime": "2026-08-19T09:12:16.108956001Z",
      "endTime": "2026-08-19T09:31:18.830785986Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc000b74a20>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-lkv5vwt7thrp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bffdb4eb-27d1-4f47-9de6-71f2cf71c0a4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bffdb4eb-27d1-4f47-9de6-71f2cf71c0a4",
      "name": "bffdb4eb-27d1-4f47-9de6-71f2cf71c0a4",
      "status": "Failed",
      "startTime": "2026-08-19T08:25:05.918235757Z",
      "endTime": "2026-08-19T08:44:07.355934418Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc000b74a20>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-lkv5vwt7thrp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bffdb4eb-27d1-4f47-9de6-71f2cf71c0a4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bffdb4eb-27d1-4f47-9de6-71f2cf71c0a4",
      "name": "bffdb4eb-27d1-4f47-9de6-71f2cf71c0a4",
      "status": "Failed",
      "startTime": "2026-08-19T08:25:05.918235757Z",
      "endTime": "2026-08-19T08:44:07.355934418Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc000a1e180>: 
    failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-lsj89ks8nhck" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c08ef4e2-2d04-4654-af6a-07027702dc7d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c08ef4e2-2d04-4654-af6a-07027702dc7d",
      "name": "c08ef4e2-2d04-4654-af6a-07027702dc7d",
      "status": "Failed",
      "startTime": "2026-08-19T08:24:46.607850855Z",
      "endTime": "2026-08-19T08:43:47.479265376Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc000a1e180>: 
    failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-lsj89ks8nhck" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c08ef4e2-2d04-4654-af6a-07027702dc7d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c08ef4e2-2d04-4654-af6a-07027702dc7d",
      "name": "c08ef4e2-2d04-4654-af6a-07027702dc7d",
      "status": "Failed",
      "startTime": "2026-08-19T08:24:46.607850855Z",
      "endTime": "2026-08-19T08:43:47.479265376Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000982a00>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-25hpr57kw8x9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/21d37e2b-c483-446d-b059-11c7cc94b492
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/21d37e2b-c483-446d-b059-11c7cc94b492",
      "name": "21d37e2b-c483-446d-b059-11c7cc94b492",
      "status": "Failed",
      "startTime": "2026-08-19T08:24:25.817368265Z",
      "endTime": "2026-08-19T08:43:27.754349217Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000982a00>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-25hpr57kw8x9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/21d37e2b-c483-446d-b059-11c7cc94b492
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/21d37e2b-c483-446d-b059-11c7cc94b492",
      "name": "21d37e2b-c483-446d-b059-11c7cc94b492",
      "status": "Failed",
      "startTime": "2026-08-19T08:24:25.817368265Z",
      "endTime": "2026-08-19T08:43:27.754349217Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
08:22:33 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapErrors | 0xc0017b6150>: 
    failed to create HCP cluster cilium-cl, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cilium-cl in resource group cni-cilium-798z4whrn9t5, error: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-798z4whrn9t5" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cilium-cl in resource group cni-cilium-798z4whrn9t5, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapErrors | 0xc0017b6150>: 
    failed to create HCP cluster cilium-cl, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cilium-cl in resource group cni-cilium-798z4whrn9t5, error: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-798z4whrn9t5" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cilium-cl in resource group cni-cilium-798z4whrn9t5, error: context deadline exceeded
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-21-gv9hgw" with version 4.21.28 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000d9ea20>: 
    failed to create HCP cluster cluster-zstream-4-21-gv9hgw: failed waiting for cluster="cluster-zstream-4-21-gv9hgw" in resourcegroup="rg-zstream-upgrade-4-21-5tt9ll7pp27c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/72a9448e-8164-458c-aad2-98cbc2592898
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/72a9448e-8164-458c-aad2-98cbc2592898",
      "name": "72a9448e-8164-458c-aad2-98cbc2592898",
      "status": "Failed",
      "startTime": "2026-08-19T09:11:52.253118092Z",
      "endTime": "2026-08-19T09:30:58.688039675Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-21-gv9hgw" with version 4.21.28 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000d9ea20>: 
    failed to create HCP cluster cluster-zstream-4-21-gv9hgw: failed waiting for cluster="cluster-zstream-4-21-gv9hgw" in resourcegroup="rg-zstream-upgrade-4-21-5tt9ll7pp27c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/72a9448e-8164-458c-aad2-98cbc2592898
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/72a9448e-8164-458c-aad2-98cbc2592898",
      "name": "72a9448e-8164-458c-aad2-98cbc2592898",
      "status": "Failed",
      "startTime": "2026-08-19T09:11:52.253118092Z",
      "endTime": "2026-08-19T09:30:58.688039675Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260630preview and verify the ingress is internal · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc0008c0400>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-wb7d4xsjg262" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/99fb1a7e-a835-40eb-a4d1-e189261a3bff
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/99fb1a7e-a835-40eb-a4d1-e189261a3bff",
      "name": "99fb1a7e-a835-40eb-a4d1-e189261a3bff",
      "status": "Failed",
      "startTime": "2026-08-19T08:25:06.228943321Z",
      "endTime": "2026-08-19T08:44:07.475048823Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc0008c0400>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-wb7d4xsjg262" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/99fb1a7e-a835-40eb-a4d1-e189261a3bff
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/99fb1a7e-a835-40eb-a4d1-e189261a3bff",
      "name": "99fb1a7e-a835-40eb-a4d1-e189261a3bff",
      "status": "Failed",
      "startTime": "2026-08-19T08:25:06.228943321Z",
      "endTime": "2026-08-19T08:44:07.475048823Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0xc0011969a0>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-2lw7bk6zfn96" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/dd6fc603-caac-4553-8b71-08c1967cfc9c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/dd6fc603-caac-4553-8b71-08c1967cfc9c",
      "name": "dd6fc603-caac-4553-8b71-08c1967cfc9c",
      "status": "Failed",
      "startTime": "2026-08-19T08:24:36.343789963Z",
      "endTime": "2026-08-19T08:43:37.468530963Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0xc0011969a0>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-2lw7bk6zfn96" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/dd6fc603-caac-4553-8b71-08c1967cfc9c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/dd6fc603-caac-4553-8b71-08c1967cfc9c",
      "name": "dd6fc603-caac-4553-8b71-08c1967cfc9c",
      "status": "Failed",
      "startTime": "2026-08-19T08:24:36.343789963Z",
      "endTime": "2026-08-19T08:43:37.468530963Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-22-7djtpb" with version 4.22.8 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000b05200>: 
    failed to create HCP cluster cluster-zstream-4-22-7djtpb: failed waiting for cluster="cluster-zstream-4-22-7djtpb" in resourcegroup="rg-zstream-upgrade-4-22-g6tvt2sj7wps" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ee74aff1-e2c3-4961-9f01-b83f367fdefa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ee74aff1-e2c3-4961-9f01-b83f367fdefa",
      "name": "ee74aff1-e2c3-4961-9f01-b83f367fdefa",
      "status": "Failed",
      "startTime": "2026-08-19T09:12:17.308417807Z",
      "endTime": "2026-08-19T09:34:27.693316817Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-22-7djtpb" with version 4.22.8 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000b05200>: 
    failed to create HCP cluster cluster-zstream-4-22-7djtpb: failed waiting for cluster="cluster-zstream-4-22-7djtpb" in resourcegroup="rg-zstream-upgrade-4-22-g6tvt2sj7wps" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ee74aff1-e2c3-4961-9f01-b83f367fdefa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ee74aff1-e2c3-4961-9f01-b83f367fdefa",
      "name": "ee74aff1-e2c3-4961-9f01-b83f367fdefa",
      "status": "Failed",
      "startTime": "2026-08-19T09:12:17.308417807Z",
      "endTime": "2026-08-19T09:34:27.693316817Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-smkcg7 with version 4.21.29
Unexpected error:
    <*fmt.wrapError | 0xc000880c60>: 
    failed to create HCP cluster np-version-upgrade-cluster-smkcg7: failed waiting for cluster="np-version-upgrade-cluster-smkcg7" in resourcegroup="rg-np-version-upgrade-smkcg7-xtk78kmwbb6j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c5ebe098-780f-4616-bc37-8de2a13512b8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c5ebe098-780f-4616-bc37-8de2a13512b8",
      "name": "c5ebe098-780f-4616-bc37-8de2a13512b8",
      "status": "Failed",
      "startTime": "2026-08-19T09:57:47.84735501Z",
      "endTime": "2026-08-19T10:16:50.482233887Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-smkcg7 with version 4.21.29
Unexpected error:
    <*fmt.wrapError | 0xc000880c60>: 
    failed to create HCP cluster np-version-upgrade-cluster-smkcg7: failed waiting for cluster="np-version-upgrade-cluster-smkcg7" in resourcegroup="rg-np-version-upgrade-smkcg7-xtk78kmwbb6j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c5ebe098-780f-4616-bc37-8de2a13512b8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c5ebe098-780f-4616-bc37-8de2a13512b8",
      "name": "c5ebe098-780f-4616-bc37-8de2a13512b8",
      "status": "Failed",
      "startTime": "2026-08-19T09:57:47.84735501Z",
      "endTime": "2026-08-19T10:16:50.482233887Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc0004b0d80>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-727szjzq886r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/09c96767-77b8-4b54-8b3e-fd329327712a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/09c96767-77b8-4b54-8b3e-fd329327712a",
      "name": "09c96767-77b8-4b54-8b3e-fd329327712a",
      "status": "Failed",
      "startTime": "2026-08-19T09:18:51.186294031Z",
      "endTime": "2026-08-19T09:37:59.707203899Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc0004b0d80>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-727szjzq886r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/09c96767-77b8-4b54-8b3e-fd329327712a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/09c96767-77b8-4b54-8b3e-fd329327712a",
      "name": "09c96767-77b8-4b54-8b3e-fd329327712a",
      "status": "Failed",
      "startTime": "2026-08-19T09:18:51.186294031Z",
      "endTime": "2026-08-19T09:37:59.707203899Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000efe640>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-2mstjmrnrrbc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/25cb657c-dfcb-42cb-95e9-754919eefb2c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/25cb657c-dfcb-42cb-95e9-754919eefb2c",
      "name": "25cb657c-dfcb-42cb-95e9-754919eefb2c",
      "status": "Failed",
      "startTime": "2026-08-19T09:16:45.942287667Z",
      "endTime": "2026-08-19T09:35:49.808663057Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000efe640>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-2mstjmrnrrbc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/25cb657c-dfcb-42cb-95e9-754919eefb2c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/25cb657c-dfcb-42cb-95e9-754919eefb2c",
      "name": "25cb657c-dfcb-42cb-95e9-754919eefb2c",
      "status": "Failed",
      "startTime": "2026-08-19T09:16:45.942287667Z",
      "endTime": "2026-08-19T09:35:49.808663057Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc000219ae0>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-47vsjx8txmh7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a1f01db-9da2-40fb-b8a3-89761501b43e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a1f01db-9da2-40fb-b8a3-89761501b43e",
      "name": "0a1f01db-9da2-40fb-b8a3-89761501b43e",
      "status": "Failed",
      "startTime": "2026-08-19T08:30:19.20307522Z",
      "endTime": "2026-08-19T08:49:27.667911825Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc000219ae0>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-47vsjx8txmh7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a1f01db-9da2-40fb-b8a3-89761501b43e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a1f01db-9da2-40fb-b8a3-89761501b43e",
      "name": "0a1f01db-9da2-40fb-b8a3-89761501b43e",
      "status": "Failed",
      "startTime": "2026-08-19T08:30:19.20307522Z",
      "endTime": "2026-08-19T08:49:27.667911825Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:217]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0009cf3e0>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-fdtxqfw26w8k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b3b51041-85b1-485d-94fb-98aec3514c12
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b3b51041-85b1-485d-94fb-98aec3514c12",
      "name": "b3b51041-85b1-485d-94fb-98aec3514c12",
      "status": "Failed",
      "startTime": "2026-08-19T09:58:46.685117867Z",
      "endTime": "2026-08-19T10:17:50.428351161Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:217]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0009cf3e0>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-fdtxqfw26w8k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b3b51041-85b1-485d-94fb-98aec3514c12
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b3b51041-85b1-485d-94fb-98aec3514c12",
      "name": "b3b51041-85b1-485d-94fb-98aec3514c12",
      "status": "Failed",
      "startTime": "2026-08-19T09:58:46.685117867Z",
      "endTime": "2026-08-19T10:17:50.428351161Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
cluster entered terminal Failed state after role assignment deployment
08:22:33 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000dddaa0>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-cr9cfng8xj4h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/387da12f-379d-4368-b932-a01949c5f0f3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/387da12f-379d-4368-b932-a01949c5f0f3",
      "name": "387da12f-379d-4368-b932-a01949c5f0f3",
      "status": "Failed",
      "startTime": "2026-08-19T09:15:20.235090894Z",
      "endTime": "2026-08-19T09:34:28.897849477Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000dddaa0>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-cr9cfng8xj4h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/387da12f-379d-4368-b932-a01949c5f0f3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/387da12f-379d-4368-b932-a01949c5f0f3",
      "name": "387da12f-379d-4368-b932-a01949c5f0f3",
      "status": "Failed",
      "startTime": "2026-08-19T09:15:20.235090894Z",
      "endTime": "2026-08-19T09:34:28.897849477Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc00056f000>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-fhtzqbfv8ljs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5cd7a0af-825d-4f09-b915-937e3e806442
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5cd7a0af-825d-4f09-b915-937e3e806442",
      "name": "5cd7a0af-825d-4f09-b915-937e3e806442",
      "status": "Failed",
      "startTime": "2026-08-19T08:27:27.193980935Z",
      "endTime": "2026-08-19T08:46:27.796618416Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc00056f000>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-fhtzqbfv8ljs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5cd7a0af-825d-4f09-b915-937e3e806442
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5cd7a0af-825d-4f09-b915-937e3e806442",
      "name": "5cd7a0af-825d-4f09-b915-937e3e806442",
      "status": "Failed",
      "startTime": "2026-08-19T08:27:27.193980935Z",
      "endTime": "2026-08-19T08:46:27.796618416Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000a8ace0>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-jc8w2p57bbv7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8fba23c0-2ad3-43d9-9019-418078e76952
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8fba23c0-2ad3-43d9-9019-418078e76952",
      "name": "8fba23c0-2ad3-43d9-9019-418078e76952",
      "status": "Failed",
      "startTime": "2026-08-19T09:09:26.073776292Z",
      "endTime": "2026-08-19T09:28:28.94049726Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000a8ace0>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-jc8w2p57bbv7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8fba23c0-2ad3-43d9-9019-418078e76952
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8fba23c0-2ad3-43d9-9019-418078e76952",
      "name": "8fba23c0-2ad3-43d9-9019-418078e76952",
      "status": "Failed",
      "startTime": "2026-08-19T09:09:26.073776292Z",
      "endTime": "2026-08-19T09:28:28.94049726Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:462]: failed to create HCP cluster np-skip-minor-vbp599 at version 4.22.10
Unexpected error:
    <*fmt.wrapError | 0xc00119a300>: 
    failed to create HCP cluster np-skip-minor-vbp599: failed waiting for cluster="np-skip-minor-vbp599" in resourcegroup="rg-np-skip-minor-vbp599-tgssghs7h24s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d0b49569-5ef9-458e-a72f-349fe2cedcca
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d0b49569-5ef9-458e-a72f-349fe2cedcca",
      "name": "d0b49569-5ef9-458e-a72f-349fe2cedcca",
      "status": "Failed",
      "startTime": "2026-08-19T09:57:27.478432272Z",
      "endTime": "2026-08-19T10:16:30.510420634Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:462]: failed to create HCP cluster np-skip-minor-vbp599 at version 4.22.10
Unexpected error:
    <*fmt.wrapError | 0xc00119a300>: 
    failed to create HCP cluster np-skip-minor-vbp599: failed waiting for cluster="np-skip-minor-vbp599" in resourcegroup="rg-np-skip-minor-vbp599-tgssghs7h24s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d0b49569-5ef9-458e-a72f-349fe2cedcca
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d0b49569-5ef9-458e-a72f-349fe2cedcca",
      "name": "d0b49569-5ef9-458e-a72f-349fe2cedcca",
      "status": "Failed",
      "startTime": "2026-08-19T09:57:27.478432272Z",
      "endTime": "2026-08-19T10:16:30.510420634Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000ec4be0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-xjvmstvh4btm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4adc6f9c-5eca-4059-b1b1-8077c0f97a68
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4adc6f9c-5eca-4059-b1b1-8077c0f97a68",
      "name": "4adc6f9c-5eca-4059-b1b1-8077c0f97a68",
      "status": "Failed",
      "startTime": "2026-08-19T09:15:05.820355787Z",
      "endTime": "2026-08-19T09:34:08.843443312Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000ec4be0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-xjvmstvh4btm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4adc6f9c-5eca-4059-b1b1-8077c0f97a68
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4adc6f9c-5eca-4059-b1b1-8077c0f97a68",
      "name": "4adc6f9c-5eca-4059-b1b1-8077c0f97a68",
      "status": "Failed",
      "startTime": "2026-08-19T09:15:05.820355787Z",
      "endTime": "2026-08-19T09:34:08.843443312Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:741]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0004b0680>: 
    failed to create HCP cluster np-dg-minor-8m57tx: failed waiting for cluster="np-dg-minor-8m57tx" in resourcegroup="rg-np-dg-minor-8m57tx-xdkz692f8vd7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b7a9db34-6c5b-44f9-8a6f-d33e4e3094c1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b7a9db34-6c5b-44f9-8a6f-d33e4e3094c1",
      "name": "b7a9db34-6c5b-44f9-8a6f-d33e4e3094c1",
      "status": "Failed",
      "startTime": "2026-08-19T09:58:03.297575836Z",
      "endTime": "2026-08-19T10:17:10.529991477Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:741]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0004b0680>: 
    failed to create HCP cluster np-dg-minor-8m57tx: failed waiting for cluster="np-dg-minor-8m57tx" in resourcegroup="rg-np-dg-minor-8m57tx-xdkz692f8vd7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b7a9db34-6c5b-44f9-8a6f-d33e4e3094c1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b7a9db34-6c5b-44f9-8a6f-d33e4e3094c1",
      "name": "b7a9db34-6c5b-44f9-8a6f-d33e4e3094c1",
      "status": "Failed",
      "startTime": "2026-08-19T09:58:03.297575836Z",
      "endTime": "2026-08-19T10:17:10.529991477Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-20-whf99d" with version 4.20.33 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000f889c0>: 
    failed to create HCP cluster cluster-zstream-4-20-whf99d: failed waiting for cluster="cluster-zstream-4-20-whf99d" in resourcegroup="rg-zstream-upgrade-4-20-jnxmc6l6ws8s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8f1aff62-695b-49ea-987e-f070233cb82c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8f1aff62-695b-49ea-987e-f070233cb82c",
      "name": "8f1aff62-695b-49ea-987e-f070233cb82c",
      "status": "Failed",
      "startTime": "2026-08-19T09:11:36.143444334Z",
      "endTime": "2026-08-19T09:30:38.762016016Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-20-whf99d" with version 4.20.33 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000f889c0>: 
    failed to create HCP cluster cluster-zstream-4-20-whf99d: failed waiting for cluster="cluster-zstream-4-20-whf99d" in resourcegroup="rg-zstream-upgrade-4-20-jnxmc6l6ws8s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8f1aff62-695b-49ea-987e-f070233cb82c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8f1aff62-695b-49ea-987e-f070233cb82c",
      "name": "8f1aff62-695b-49ea-987e-f070233cb82c",
      "status": "Failed",
      "startTime": "2026-08-19T09:11:36.143444334Z",
      "endTime": "2026-08-19T09:30:38.762016016Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
08:22:33 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-5-0-pxp4bpz828sp/cluster-candidate-5-0-cpprpm should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc001990630>: 
    failed to create HCP cluster cluster-candidate-5-0-cpprpm, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-cpprpm in resource group rg-candidate-5-0-pxp4bpz828sp, error: failed waiting for cluster="cluster-candidate-5-0-cpprpm" in resourcegroup="rg-candidate-5-0-pxp4bpz828sp" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-cpprpm in resource group rg-candidate-5-0-pxp4bpz828sp, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-5-0-pxp4bpz828sp/cluster-candidate-5-0-cpprpm should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc001990630>: 
    failed to create HCP cluster cluster-candidate-5-0-cpprpm, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-cpprpm in resource group rg-candidate-5-0-pxp4bpz828sp, error: failed waiting for cluster="cluster-candidate-5-0-cpprpm" in resourcegroup="rg-candidate-5-0-pxp4bpz828sp" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-cpprpm in resource group rg-candidate-5-0-pxp4bpz828sp, error: context deadline exceeded
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000957280>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-pt2r4dggcq5r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bb5892dc-0282-4451-aacb-7e1a4917e44f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bb5892dc-0282-4451-aacb-7e1a4917e44f",
      "name": "bb5892dc-0282-4451-aacb-7e1a4917e44f",
      "status": "Failed",
      "startTime": "2026-08-19T08:26:10.546101333Z",
      "endTime": "2026-08-19T08:45:17.499297631Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000957280>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-pt2r4dggcq5r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bb5892dc-0282-4451-aacb-7e1a4917e44f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bb5892dc-0282-4451-aacb-7e1a4917e44f",
      "name": "bb5892dc-0282-4451-aacb-7e1a4917e44f",
      "status": "Failed",
      "startTime": "2026-08-19T08:26:10.546101333Z",
      "endTime": "2026-08-19T08:45:17.499297631Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
08:22:33 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:151]: failed to create HCP cluster version 4.19
Unexpected error:
    <*fmt.wrapErrors | 0xc001604900>: 
    failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-f6cftzktxxjd" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-f6cftzktxxjd, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:151]: failed to create HCP cluster version 4.19
Unexpected error:
    <*fmt.wrapErrors | 0xc001604900>: 
    failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-f6cftzktxxjd" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-f6cftzktxxjd, error: context deadline exceeded
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-nd76dl with version 4.21.29
Unexpected error:
    <*fmt.wrapError | 0xc000ed8f20>: 
    failed to create HCP cluster np-version-upgrade-cluster-nd76dl: failed waiting for cluster="np-version-upgrade-cluster-nd76dl" in resourcegroup="rg-np-version-upgrade-nd76dl-tm2h7sjghrqf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fc2acd98-147f-4034-9860-21b14fd50cff
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fc2acd98-147f-4034-9860-21b14fd50cff",
      "name": "fc2acd98-147f-4034-9860-21b14fd50cff",
      "status": "Failed",
      "startTime": "2026-08-19T09:56:12.73063123Z",
      "endTime": "2026-08-19T10:15:20.43403027Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-nd76dl with version 4.21.29
Unexpected error:
    <*fmt.wrapError | 0xc000ed8f20>: 
    failed to create HCP cluster np-version-upgrade-cluster-nd76dl: failed waiting for cluster="np-version-upgrade-cluster-nd76dl" in resourcegroup="rg-np-version-upgrade-nd76dl-tm2h7sjghrqf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fc2acd98-147f-4034-9860-21b14fd50cff
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fc2acd98-147f-4034-9860-21b14fd50cff",
      "name": "fc2acd98-147f-4034-9860-21b14fd50cff",
      "status": "Failed",
      "startTime": "2026-08-19T09:56:12.73063123Z",
      "endTime": "2026-08-19T10:15:20.43403027Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: Conflict; detail message Cannot update resource while resource is "provisioning"; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:314]: failed to create HCP cluster np-noedge-chjtcz
Unexpected error:
    <*fmt.wrapError | 0xc000fcc1e0>: 
    failed to create HCP cluster np-noedge-chjtcz: failed starting cluster creation "np-noedge-chjtcz" in resourcegroup="rg-np-noedge-chjtcz-4lgwg5nz5hfm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-chjtcz-4lgwg5nz5hfm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-chjtcz
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot update resource while resource is \"provisioning\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-chjtcz-4lgwg5nz5hfm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-chjtcz"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:314]: failed to create HCP cluster np-noedge-chjtcz
Unexpected error:
    <*fmt.wrapError | 0xc000fcc1e0>: 
    failed to create HCP cluster np-noedge-chjtcz: failed starting cluster creation "np-noedge-chjtcz" in resourcegroup="rg-np-noedge-chjtcz-4lgwg5nz5hfm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-chjtcz-4lgwg5nz5hfm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-chjtcz
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot update resource while resource is \"provisioning\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-chjtcz-4lgwg5nz5hfm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-chjtcz"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Cluster should become ready within '20.000000' minutes
08:22:33 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:163]: Timed out after 1200.003s.
Cluster should become ready within '20.000000' minutes
Expected
    <bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:163]: Timed out after 1200.003s.
Cluster should become ready within '20.000000' minutes
Expected
    <bool>: ...
to be true
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '20.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc001284db0>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e12d2654-0454-40c0-b3c4-ba86f729b72b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e12d2654-0454-40c0-b3c4-ba86f729b72b",
      "name": "e12d2654-0454-40c0-b3c4-ba86f729b72b",
      "status": "Failed",
      "startTime": "2026-08-19T08:31:23.522024844Z",
      "endTime": "2026-08-19T08:50:27.682907631Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '20.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc001284db0>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e12d2654-0454-40c0-b3c4-ba86f729b72b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e12d2654-0454-40c0-b3c4-ba86f729b72b",
      "name": "e12d2654-0454-40c0-b3c4-ba86f729b72b",
      "status": "Failed",
      "startTime": "2026-08-19T08:31:23.522024844Z",
      "endTime": "2026-08-19T08:50:27.682907631Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000588ae0>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-q8slhldwd8fs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fd3b0752-16e1-4140-8c09-4ea1166fe507
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fd3b0752-16e1-4140-8c09-4ea1166fe507",
      "name": "fd3b0752-16e1-4140-8c09-4ea1166fe507",
      "status": "Failed",
      "startTime": "2026-08-19T08:31:25.477632893Z",
      "endTime": "2026-08-19T08:50:27.950759362Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000588ae0>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-q8slhldwd8fs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fd3b0752-16e1-4140-8c09-4ea1166fe507
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fd3b0752-16e1-4140-8c09-4ea1166fe507",
      "name": "fd3b0752-16e1-4140-8c09-4ea1166fe507",
      "status": "Failed",
      "startTime": "2026-08-19T08:31:25.477632893Z",
      "endTime": "2026-08-19T08:50:27.950759362Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-4-20-7l2l4qn9vw54/cluster-candidate-4-20-kc9blr should provision
Unexpected error:
    <*fmt.wrapError | 0xc000e7ad00>: 
    failed to create HCP cluster cluster-candidate-4-20-kc9blr: failed waiting for cluster="cluster-candidate-4-20-kc9blr" in resourcegroup="rg-candidate-4-20-7l2l4qn9vw54" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/41a4012d-334f-4192-99da-7822a9318510
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/41a4012d-334f-4192-99da-7822a9318510",
      "name": "41a4012d-334f-4192-99da-7822a9318510",
      "status": "Failed",
      "startTime": "2026-08-19T09:09:39.803541721Z",
      "endTime": "2026-08-19T09:28:48.770343717Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-4-20-7l2l4qn9vw54/cluster-candidate-4-20-kc9blr should provision
Unexpected error:
    <*fmt.wrapError | 0xc000e7ad00>: 
    failed to create HCP cluster cluster-candidate-4-20-kc9blr: failed waiting for cluster="cluster-candidate-4-20-kc9blr" in resourcegroup="rg-candidate-4-20-7l2l4qn9vw54" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/41a4012d-334f-4192-99da-7822a9318510
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/41a4012d-334f-4192-99da-7822a9318510",
      "name": "41a4012d-334f-4192-99da-7822a9318510",
      "status": "Failed",
      "startTime": "2026-08-19T09:09:39.803541721Z",
      "endTime": "2026-08-19T09:28:48.770343717Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000f44300>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-vx74sj5tq57q" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2369450e-ba00-460b-8979-1eb3414f2f56
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2369450e-ba00-460b-8979-1eb3414f2f56",
      "name": "2369450e-ba00-460b-8979-1eb3414f2f56",
      "status": "Failed",
      "startTime": "2026-08-19T08:26:59.984487987Z",
      "endTime": "2026-08-19T08:46:07.563867327Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000f44300>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-vx74sj5tq57q" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2369450e-ba00-460b-8979-1eb3414f2f56
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2369450e-ba00-460b-8979-1eb3414f2f56",
      "name": "2369450e-ba00-460b-8979-1eb3414f2f56",
      "status": "Failed",
      "startTime": "2026-08-19T08:26:59.984487987Z",
      "endTime": "2026-08-19T08:46:07.563867327Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc0004b05e0>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-nb94ssj78t8b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8114585d-7355-4a77-bc6b-a3cb85fc2906
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8114585d-7355-4a77-bc6b-a3cb85fc2906",
      "name": "8114585d-7355-4a77-bc6b-a3cb85fc2906",
      "status": "Failed",
      "startTime": "2026-08-19T09:13:20.223750233Z",
      "endTime": "2026-08-19T09:32:28.958351882Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc0004b05e0>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-nb94ssj78t8b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8114585d-7355-4a77-bc6b-a3cb85fc2906
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8114585d-7355-4a77-bc6b-a3cb85fc2906",
      "name": "8114585d-7355-4a77-bc6b-a3cb85fc2906",
      "status": "Failed",
      "startTime": "2026-08-19T09:13:20.223750233Z",
      "endTime": "2026-08-19T09:32:28.958351882Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
08:22:33 UTC · failed at e2e · test Customer should be able to create an HCP cluster then delete it by deleting the customer resource group · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc000937110>: 
    failed to create HCP cluster cx-rg-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cx-rg-hcp-cluster in resource group cx-rg-cluster-nlpcprvgpdvf, error: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-nlpcprvgpdvf" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cx-rg-hcp-cluster in resource group cx-rg-cluster-nlpcprvgpdvf, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc000937110>: 
    failed to create HCP cluster cx-rg-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cx-rg-hcp-cluster in resource group cx-rg-cluster-nlpcprvgpdvf, error: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-nlpcprvgpdvf" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cx-rg-hcp-cluster in resource group cx-rg-cluster-nlpcprvgpdvf, error: context deadline exceeded
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:98]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc000a9c680>: 
    failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-pp2n8n82cnm8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ccaae1be-9bd2-4d98-9326-179cb4972328
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ccaae1be-9bd2-4d98-9326-179cb4972328",
      "name": "ccaae1be-9bd2-4d98-9326-179cb4972328",
      "status": "Failed",
      "startTime": "2026-08-19T09:27:41.137415329Z",
      "endTime": "2026-08-19T09:46:49.779332462Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:98]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc000a9c680>: 
    failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-pp2n8n82cnm8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ccaae1be-9bd2-4d98-9326-179cb4972328
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ccaae1be-9bd2-4d98-9326-179cb4972328",
      "name": "ccaae1be-9bd2-4d98-9326-179cb4972328",
      "status": "Failed",
      "startTime": "2026-08-19T09:27:41.137415329Z",
      "endTime": "2026-08-19T09:46:49.779332462Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc000db2b00>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-2pw62s54wb2g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2fa6ff02-0f21-4b43-8ea3-734daeb76057
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2fa6ff02-0f21-4b43-8ea3-734daeb76057",
      "name": "2fa6ff02-0f21-4b43-8ea3-734daeb76057",
      "status": "Failed",
      "startTime": "2026-08-19T08:25:35.174793366Z",
      "endTime": "2026-08-19T08:44:37.4843169Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc000db2b00>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-2pw62s54wb2g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2fa6ff02-0f21-4b43-8ea3-734daeb76057
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2fa6ff02-0f21-4b43-8ea3-734daeb76057",
      "name": "2fa6ff02-0f21-4b43-8ea3-734daeb76057",
      "status": "Failed",
      "startTime": "2026-08-19T08:25:35.174793366Z",
      "endTime": "2026-08-19T08:44:37.4843169Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:270]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000a984e0>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-jhtq7p97bpp8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/86c9a634-3402-4cfc-aac0-0649d28f7bcc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/86c9a634-3402-4cfc-aac0-0649d28f7bcc",
      "name": "86c9a634-3402-4cfc-aac0-0649d28f7bcc",
      "status": "Failed",
      "startTime": "2026-08-19T09:58:24.882021093Z",
      "endTime": "2026-08-19T10:17:30.539305666Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:270]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000a984e0>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-jhtq7p97bpp8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/86c9a634-3402-4cfc-aac0-0649d28f7bcc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/86c9a634-3402-4cfc-aac0-0649d28f7bcc",
      "name": "86c9a634-3402-4cfc-aac0-0649d28f7bcc",
      "status": "Failed",
      "startTime": "2026-08-19T09:58:24.882021093Z",
      "endTime": "2026-08-19T10:17:30.539305666Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-19-d4gk6z" with version 4.19.41 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc00108e420>: 
    failed to create HCP cluster cluster-zstream-4-19-d4gk6z: failed waiting for cluster="cluster-zstream-4-19-d4gk6z" in resourcegroup="rg-zstream-upgrade-4-19-vztw56sqtm2k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f6df6728-abcc-4c1d-ad94-f8fc0e78e56a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f6df6728-abcc-4c1d-ad94-f8fc0e78e56a",
      "name": "f6df6728-abcc-4c1d-ad94-f8fc0e78e56a",
      "status": "Failed",
      "startTime": "2026-08-19T09:11:20.483309258Z",
      "endTime": "2026-08-19T09:30:28.813133216Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-19-d4gk6z" with version 4.19.41 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc00108e420>: 
    failed to create HCP cluster cluster-zstream-4-19-d4gk6z: failed waiting for cluster="cluster-zstream-4-19-d4gk6z" in resourcegroup="rg-zstream-upgrade-4-19-vztw56sqtm2k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f6df6728-abcc-4c1d-ad94-f8fc0e78e56a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f6df6728-abcc-4c1d-ad94-f8fc0e78e56a",
      "name": "f6df6728-abcc-4c1d-ad94-f8fc0e78e56a",
      "status": "Failed",
      "startTime": "2026-08-19T09:11:20.483309258Z",
      "endTime": "2026-08-19T09:30:28.813133216Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
08:22:33 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-4-22-7z5ghb47hkdz/cluster-candidate-4-22-mh5jtc should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc00057e5d0>: 
    failed to create HCP cluster cluster-candidate-4-22-mh5jtc, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-22-mh5jtc in resource group rg-candidate-4-22-7z5ghb47hkdz, error: failed waiting for cluster="cluster-candidate-4-22-mh5jtc" in resourcegroup="rg-candidate-4-22-7z5ghb47hkdz" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-22-mh5jtc in resource group rg-candidate-4-22-7z5ghb47hkdz, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-4-22-7z5ghb47hkdz/cluster-candidate-4-22-mh5jtc should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc00057e5d0>: 
    failed to create HCP cluster cluster-candidate-4-22-mh5jtc, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-22-mh5jtc in resource group rg-candidate-4-22-7z5ghb47hkdz, error: failed waiting for cluster="cluster-candidate-4-22-mh5jtc" in resourcegroup="rg-candidate-4-22-7z5ghb47hkdz" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-22-mh5jtc in resource group rg-candidate-4-22-7z5ghb47hkdz, error: context deadline exceeded
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:182]: failed to create HCP cluster "cp-ystream-upgrade-4-21-wh4d6d" with install version 4.20.34
Unexpected error:
    <*fmt.wrapError | 0xc000f13300>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-wh4d6d: failed waiting for cluster="cp-ystream-upgrade-4-21-wh4d6d" in resourcegroup="rg-cp-ystream-upgrade-4-21-jxk96xvwxvvc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15ccb98b-5e2b-47b1-86e9-30fb0729f99b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15ccb98b-5e2b-47b1-86e9-30fb0729f99b",
      "name": "15ccb98b-5e2b-47b1-86e9-30fb0729f99b",
      "status": "Failed",
      "startTime": "2026-08-19T09:16:07.820643421Z",
      "endTime": "2026-08-19T09:35:08.723020497Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:182]: failed to create HCP cluster "cp-ystream-upgrade-4-21-wh4d6d" with install version 4.20.34
Unexpected error:
    <*fmt.wrapError | 0xc000f13300>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-wh4d6d: failed waiting for cluster="cp-ystream-upgrade-4-21-wh4d6d" in resourcegroup="rg-cp-ystream-upgrade-4-21-jxk96xvwxvvc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15ccb98b-5e2b-47b1-86e9-30fb0729f99b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15ccb98b-5e2b-47b1-86e9-30fb0729f99b",
      "name": "15ccb98b-5e2b-47b1-86e9-30fb0729f99b",
      "status": "Failed",
      "startTime": "2026-08-19T09:16:07.820643421Z",
      "endTime": "2026-08-19T09:35:08.723020497Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000992f60>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-dkhzcxchpmbs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/27fc2bfe-285f-454f-ac04-4bf940cfdd09
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/27fc2bfe-285f-454f-ac04-4bf940cfdd09",
      "name": "27fc2bfe-285f-454f-ac04-4bf940cfdd09",
      "status": "Failed",
      "startTime": "2026-08-19T08:27:22.100052158Z",
      "endTime": "2026-08-19T08:46:27.746066667Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000992f60>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-dkhzcxchpmbs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/27fc2bfe-285f-454f-ac04-4bf940cfdd09
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/27fc2bfe-285f-454f-ac04-4bf940cfdd09",
      "name": "27fc2bfe-285f-454f-ac04-4bf940cfdd09",
      "status": "Failed",
      "startTime": "2026-08-19T08:27:22.100052158Z",
      "endTime": "2026-08-19T08:46:27.746066667Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
08:22:33 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapErrors | 0xc00075cd50>: 
    failed to create HCP cluster basic-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-xdjvrgwvf7dw, error: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-xdjvrgwvf7dw" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-xdjvrgwvf7dw, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapErrors | 0xc00075cd50>: 
    failed to create HCP cluster basic-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-xdjvrgwvf7dw, error: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-xdjvrgwvf7dw" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-xdjvrgwvf7dw, error: context deadline exceeded
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
08:22:33 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-mmxpxx with version 4.20.34
Unexpected error:
    <*fmt.wrapError | 0xc001318d00>: 
    failed to create HCP cluster np-version-upgrade-cluster-mmxpxx: failed waiting for cluster="np-version-upgrade-cluster-mmxpxx" in resourcegroup="rg-np-version-upgrade-mmxpxx-2kxn7tmk8lnt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/87c17e6f-54d2-438f-9d54-bbd25080551c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/87c17e6f-54d2-438f-9d54-bbd25080551c",
      "name": "87c17e6f-54d2-438f-9d54-bbd25080551c",
      "status": "Failed",
      "startTime": "2026-08-19T09:58:39.15919512Z",
      "endTime": "2026-08-19T10:17:40.485904285Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-mmxpxx with version 4.20.34
Unexpected error:
    <*fmt.wrapError | 0xc001318d00>: 
    failed to create HCP cluster np-version-upgrade-cluster-mmxpxx: failed waiting for cluster="np-version-upgrade-cluster-mmxpxx" in resourcegroup="rg-np-version-upgrade-mmxpxx-2kxn7tmk8lnt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/87c17e6f-54d2-438f-9d54-bbd25080551c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/87c17e6f-54d2-438f-9d54-bbd25080551c",
      "name": "87c17e6f-54d2-438f-9d54-bbd25080551c",
      "status": "Failed",
      "startTime": "2026-08-19T09:58:39.15919512Z",
      "endTime": "2026-08-19T10:17:40.485904285Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 7
e2e (7)
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift
07:04:47 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260630preview and verify the ingress is internal · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc00161c860>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-x98pjvvxgzk4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/03e64f83-5bd0-4f0b-819b-4b88633c17cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/03e64f83-5bd0-4f0b-819b-4b88633c17cf",
      "name": "03e64f83-5bd0-4f0b-819b-4b88633c17cf",
      "status": "Failed",
      "startTime": "2026-08-19T07:16:18.329159963Z",
      "endTime": "2026-08-19T07:35:23.12849681Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc00161c860>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-x98pjvvxgzk4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/03e64f83-5bd0-4f0b-819b-4b88633c17cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/03e64f83-5bd0-4f0b-819b-4b88633c17cf",
      "name": "03e64f83-5bd0-4f0b-819b-4b88633c17cf",
      "status": "Failed",
      "startTime": "2026-08-19T07:16:18.329159963Z",
      "endTime": "2026-08-19T07:35:23.12849681Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift
07:04:47 UTC · failed at e2e · test FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000f896e0>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-fh79ld4nbj75" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5bea30a8-9570-45db-b908-4c4e07deca9b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5bea30a8-9570-45db-b908-4c4e07deca9b",
      "name": "5bea30a8-9570-45db-b908-4c4e07deca9b",
      "status": "Failed",
      "startTime": "2026-08-19T07:16:30.019566976Z",
      "endTime": "2026-08-19T07:35:32.987525861Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000f896e0>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-fh79ld4nbj75" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5bea30a8-9570-45db-b908-4c4e07deca9b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5bea30a8-9570-45db-b908-4c4e07deca9b",
      "name": "5bea30a8-9570-45db-b908-4c4e07deca9b",
      "status": "Failed",
      "startTime": "2026-08-19T07:16:30.019566976Z",
      "endTime": "2026-08-19T07:35:32.987525861Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
07:04:47 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc00102e440>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-td7x448jbghw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b82c4e2a-858b-4ab5-b7da-52185c1899e8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b82c4e2a-858b-4ab5-b7da-52185c1899e8",
      "name": "b82c4e2a-858b-4ab5-b7da-52185c1899e8",
      "status": "Failed",
      "startTime": "2026-08-19T07:17:51.501004988Z",
      "endTime": "2026-08-19T07:36:53.373049008Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc00102e440>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-td7x448jbghw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b82c4e2a-858b-4ab5-b7da-52185c1899e8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b82c4e2a-858b-4ab5-b7da-52185c1899e8",
      "name": "b82c4e2a-858b-4ab5-b7da-52185c1899e8",
      "status": "Failed",
      "startTime": "2026-08-19T07:17:51.501004988Z",
      "endTime": "2026-08-19T07:36:53.373049008Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; provider Microsoft.RedHatOpenShift
07:04:47 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-4-22-zg4z5cbh25nm/cluster-candidate-4-22-tbzjpf should provision
Unexpected error:
    <*fmt.wrapError | 0xc0010dc4e0>: 
    failed to create HCP cluster cluster-candidate-4-22-tbzjpf: failed waiting for cluster="cluster-candidate-4-22-tbzjpf" in resourcegroup="rg-candidate-4-22-zg4z5cbh25nm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/52ce422b-7b70-4b90-aa52-fc3fa085301b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/52ce422b-7b70-4b90-aa52-fc3fa085301b",
      "name": "52ce422b-7b70-4b90-aa52-fc3fa085301b",
      "status": "Failed",
      "startTime": "2026-08-19T07:49:18.086270762Z",
      "endTime": "2026-08-19T08:08:24.407819096Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-4-22-zg4z5cbh25nm/cluster-candidate-4-22-tbzjpf should provision
Unexpected error:
    <*fmt.wrapError | 0xc0010dc4e0>: 
    failed to create HCP cluster cluster-candidate-4-22-tbzjpf: failed waiting for cluster="cluster-candidate-4-22-tbzjpf" in resourcegroup="rg-candidate-4-22-zg4z5cbh25nm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/52ce422b-7b70-4b90-aa52-fc3fa085301b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/52ce422b-7b70-4b90-aa52-fc3fa085301b",
      "name": "52ce422b-7b70-4b90-aa52-fc3fa085301b",
      "status": "Failed",
      "startTime": "2026-08-19T07:49:18.086270762Z",
      "endTime": "2026-08-19T08:08:24.407819096Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift
07:04:47 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc00082a120>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-ml5crr8fbhl2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eceba15d-5a1a-4b27-a4e7-b24724b60536
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eceba15d-5a1a-4b27-a4e7-b24724b60536",
      "name": "eceba15d-5a1a-4b27-a4e7-b24724b60536",
      "status": "Failed",
      "startTime": "2026-08-19T07:18:43.866677598Z",
      "endTime": "2026-08-19T07:37:43.126289194Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc00082a120>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-ml5crr8fbhl2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eceba15d-5a1a-4b27-a4e7-b24724b60536
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eceba15d-5a1a-4b27-a4e7-b24724b60536",
      "name": "eceba15d-5a1a-4b27-a4e7-b24724b60536",
      "status": "Failed",
      "startTime": "2026-08-19T07:18:43.866677598Z",
      "endTime": "2026-08-19T07:37:43.126289194Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift
07:04:47 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-4-21-dvgp2ljncmzv/cluster-candidate-4-21-m56pdm should provision
Unexpected error:
    <*fmt.wrapError | 0xc000866520>: 
    failed to create HCP cluster cluster-candidate-4-21-m56pdm: failed waiting for cluster="cluster-candidate-4-21-m56pdm" in resourcegroup="rg-candidate-4-21-dvgp2ljncmzv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc9b40e4-dff9-45e8-b9ce-61a92321163f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc9b40e4-dff9-45e8-b9ce-61a92321163f",
      "name": "dc9b40e4-dff9-45e8-b9ce-61a92321163f",
      "status": "Failed",
      "startTime": "2026-08-19T07:48:53.852097725Z",
      "endTime": "2026-08-19T08:07:54.416992123Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-4-21-dvgp2ljncmzv/cluster-candidate-4-21-m56pdm should provision
Unexpected error:
    <*fmt.wrapError | 0xc000866520>: 
    failed to create HCP cluster cluster-candidate-4-21-m56pdm: failed waiting for cluster="cluster-candidate-4-21-m56pdm" in resourcegroup="rg-candidate-4-21-dvgp2ljncmzv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc9b40e4-dff9-45e8-b9ce-61a92321163f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc9b40e4-dff9-45e8-b9ce-61a92321163f",
      "name": "dc9b40e4-dff9-45e8-b9ce-61a92321163f",
      "status": "Failed",
      "startTime": "2026-08-19T07:48:53.852097725Z",
      "endTime": "2026-08-19T08:07:54.416992123Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift
07:04:47 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000817120>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-9k28vzcm69z8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d038a282-6d26-4349-9d6e-fafe88a9f437
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d038a282-6d26-4349-9d6e-fafe88a9f437",
      "name": "d038a282-6d26-4349-9d6e-fafe88a9f437",
      "status": "Failed",
      "startTime": "2026-08-19T07:17:50.18783504Z",
      "endTime": "2026-08-19T07:36:53.392353745Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000817120>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-9k28vzcm69z8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d038a282-6d26-4349-9d6e-fafe88a9f437
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d038a282-6d26-4349-9d6e-fafe88a9f437",
      "name": "d038a282-6d26-4349-9d6e-fafe88a9f437",
      "status": "Failed",
      "startTime": "2026-08-19T07:17:50.18783504Z",
      "endTime": "2026-08-19T07:36:53.392353745Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 12
e2e (12)
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
05:29:11 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000be4420>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-skip-minor-wsd5rv" in resourcegroup="rg-np-skip-minor-wsd5rv-7zc5n446c49n" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44029b45-2a06-403c-ad77-6b306bc07eac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44029b45-2a06-403c-ad77-6b306bc07eac",
      "name": "44029b45-2a06-403c-ad77-6b306bc07eac",
      "status": "Failed",
      "startTime": "2026-08-19T07:45:10.619596164Z",
      "endTime": "2026-08-19T08:09:15.782431257Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9gasbeacg14154vgqri92i9qj3lj2u still exists (deletion dispatched at 2026-08-19T07:45:10Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000be4420>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-skip-minor-wsd5rv" in resourcegroup="rg-np-skip-minor-wsd5rv-7zc5n446c49n" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44029b45-2a06-403c-ad77-6b306bc07eac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44029b45-2a06-403c-ad77-6b306bc07eac",
      "name": "44029b45-2a06-403c-ad77-6b306bc07eac",
      "status": "Failed",
      "startTime": "2026-08-19T07:45:10.619596164Z",
      "endTime": "2026-08-19T08:09:15.782431257Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9gasbeacg14154vgqri92i9qj3lj2u still exists (deletion dispatched at 2026-08-19T07:45:10Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
05:29:11 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc00116e1b0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-dxqx7n" in resourcegroup="rg-np-version-upgrade-dxqx7n-j7h8sl7k4nkw" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6d75e2fa-2f17-4c70-b06f-914c5a7c9ff5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6d75e2fa-2f17-4c70-b06f-914c5a7c9ff5",
      "name": "6d75e2fa-2f17-4c70-b06f-914c5a7c9ff5",
      "status": "Failed",
      "startTime": "2026-08-19T07:31:53.195424705Z",
      "endTime": "2026-08-19T07:55:54.488364404Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9g68gmftv667t18n90kkemktfaa0e8 still exists (deletion dispatched at 2026-08-19T07:31:53Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc00116e1b0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-dxqx7n" in resourcegroup="rg-np-version-upgrade-dxqx7n-j7h8sl7k4nkw" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6d75e2fa-2f17-4c70-b06f-914c5a7c9ff5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6d75e2fa-2f17-4c70-b06f-914c5a7c9ff5",
      "name": "6d75e2fa-2f17-4c70-b06f-914c5a7c9ff5",
      "status": "Failed",
      "startTime": "2026-08-19T07:31:53.195424705Z",
      "endTime": "2026-08-19T07:55:54.488364404Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9g68gmftv667t18n90kkemktfaa0e8 still exists (deletion dispatched at 2026-08-19T07:31:53Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
05:29:11 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000cdc8e8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-85gsqcc2zjlm" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/102c7ad9-cfd7-4fbb-b4af-47ae501c8b59
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/102c7ad9-cfd7-4fbb-b4af-47ae501c8b59",
      "name": "102c7ad9-cfd7-4fbb-b4af-47ae501c8b59",
      "status": "Failed",
      "startTime": "2026-08-19T07:41:39.807626461Z",
      "endTime": "2026-08-19T08:05:45.761898963Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9geiu25jubct6f4b5k5ucmu6fmf5pl still exists (deletion dispatched at 2026-08-19T07:41:39Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000cdc8e8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-85gsqcc2zjlm" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/102c7ad9-cfd7-4fbb-b4af-47ae501c8b59
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/102c7ad9-cfd7-4fbb-b4af-47ae501c8b59",
      "name": "102c7ad9-cfd7-4fbb-b4af-47ae501c8b59",
      "status": "Failed",
      "startTime": "2026-08-19T07:41:39.807626461Z",
      "endTime": "2026-08-19T08:05:45.761898963Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9geiu25jubct6f4b5k5ucmu6fmf5pl still exists (deletion dispatched at 2026-08-19T07:41:39Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
05:29:11 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0011f0378>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-dw5mg9lsjg9t" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/52d1d0b5-d84a-422f-adff-632e7b15bd86
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/52d1d0b5-d84a-422f-adff-632e7b15bd86",
      "name": "52d1d0b5-d84a-422f-adff-632e7b15bd86",
      "status": "Failed",
      "startTime": "2026-08-19T07:21:20.266749938Z",
      "endTime": "2026-08-19T07:45:24.43604152Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9g587oc7gqeeckl8k5s1283accvgo6 still exists (deletion dispatched at 2026-08-19T07:21:20Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0011f0378>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-dw5mg9lsjg9t" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/52d1d0b5-d84a-422f-adff-632e7b15bd86
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/52d1d0b5-d84a-422f-adff-632e7b15bd86",
      "name": "52d1d0b5-d84a-422f-adff-632e7b15bd86",
      "status": "Failed",
      "startTime": "2026-08-19T07:21:20.266749938Z",
      "endTime": "2026-08-19T07:45:24.43604152Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9g587oc7gqeeckl8k5s1283accvgo6 still exists (deletion dispatched at 2026-08-19T07:21:20Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
05:29:11 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc00003e660>: 
    failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-kbzvplqhrp9b, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-kbzvplqhrp9b" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-kbzvplqhrp9b, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc00003e660>: 
    failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-kbzvplqhrp9b, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-kbzvplqhrp9b" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-kbzvplqhrp9b, error: context deadline exceeded
    ...
occurred
DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded
05:29:11 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000cdc768>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="oidc-wi-4m2nc64cmj8g", caused by: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-4m2nc64cmj8g" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-4m2nc64cmj8g, error: context deadline exceeded, error: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-4m2nc64cmj8g" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-4m2nc64cmj8g, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000cdc768>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="oidc-wi-4m2nc64cmj8g", caused by: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-4m2nc64cmj8g" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-4m2nc64cmj8g, error: context deadline exceeded, error: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-4m2nc64cmj8g" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-4m2nc64cmj8g, error: context deadline exceeded
    ...
occurred
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
05:29:11 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000abefa8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-noedge-9b6mxq" in resourcegroup="rg-np-noedge-9b6mxq-8gg2s6c8ddr4" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fc763824-a7c1-4c84-ad4a-52a04db780c3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fc763824-a7c1-4c84-ad4a-52a04db780c3",
      "name": "fc763824-a7c1-4c84-ad4a-52a04db780c3",
      "status": "Failed",
      "startTime": "2026-08-19T07:11:47.482159856Z",
      "endTime": "2026-08-19T07:35:54.369742847Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9g7tdod2lgoa5os2e9n6onp6vt2m2l still exists (deletion dispatched at 2026-08-19T07:11:48Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000abefa8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-noedge-9b6mxq" in resourcegroup="rg-np-noedge-9b6mxq-8gg2s6c8ddr4" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fc763824-a7c1-4c84-ad4a-52a04db780c3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fc763824-a7c1-4c84-ad4a-52a04db780c3",
      "name": "fc763824-a7c1-4c84-ad4a-52a04db780c3",
      "status": "Failed",
      "startTime": "2026-08-19T07:11:47.482159856Z",
      "endTime": "2026-08-19T07:35:54.369742847Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9g7tdod2lgoa5os2e9n6onp6vt2m2l still exists (deletion dispatched at 2026-08-19T07:11:48Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: DeploymentFailed; detail code OK
05:29:11 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:79]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000a92d60>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-disabled-image-registry-hcp-cluster-n784sz" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-disabled-image-registry-hcp-cluster-n784sz/operationStatuses/08584144855033284603
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "OK",
            "message": ""
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:79]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000a92d60>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-disabled-image-registry-hcp-cluster-n784sz" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-disabled-image-registry-hcp-cluster-n784sz/operationStatuses/08584144855033284603
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "OK",
            "message": ""
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded
05:29:11 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000f46198>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="autoscaling-cluster-rm9wbpvfcdwm", caused by: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-rm9wbpvfcdwm" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-rm9wbpvfcdwm, error: context deadline exceeded, error: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-rm9wbpvfcdwm" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-rm9wbpvfcdwm, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000f46198>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="autoscaling-cluster-rm9wbpvfcdwm", caused by: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-rm9wbpvfcdwm" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-rm9wbpvfcdwm, error: context deadline exceeded, error: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-rm9wbpvfcdwm" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-rm9wbpvfcdwm, error: context deadline exceeded
    ...
occurred
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
05:29:11 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000f06630>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-dg-minor-zm2mgg" in resourcegroup="rg-np-dg-minor-zm2mgg-fwcclg2fsc25" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/655cfcac-fbdc-4612-9646-a9b1d1f8165f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/655cfcac-fbdc-4612-9646-a9b1d1f8165f",
      "name": "655cfcac-fbdc-4612-9646-a9b1d1f8165f",
      "status": "Failed",
      "startTime": "2026-08-19T07:45:55.316555912Z",
      "endTime": "2026-08-19T08:09:55.778621843Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9ge0h765335a8886qbg3l13s24maom still exists (deletion dispatched at 2026-08-19T07:45:55Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000f06630>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-dg-minor-zm2mgg" in resourcegroup="rg-np-dg-minor-zm2mgg-fwcclg2fsc25" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/655cfcac-fbdc-4612-9646-a9b1d1f8165f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/655cfcac-fbdc-4612-9646-a9b1d1f8165f",
      "name": "655cfcac-fbdc-4612-9646-a9b1d1f8165f",
      "status": "Failed",
      "startTime": "2026-08-19T07:45:55.316555912Z",
      "endTime": "2026-08-19T08:09:55.778621843Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9ge0h765335a8886qbg3l13s24maom still exists (deletion dispatched at 2026-08-19T07:45:55Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
05:29:11 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260630preview and verify the ingress is internal · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapErrors | 0xc00073d6b0>: 
    failed to create HCP cluster private-ingress, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20260630FromParam for cluster private-ingress in resource group private-ingress-cl5k8g5wcstk, error: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-cl5k8g5wcstk" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20260630FromParam for cluster private-ingress in resource group private-ingress-cl5k8g5wcstk, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapErrors | 0xc00073d6b0>: 
    failed to create HCP cluster private-ingress, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20260630FromParam for cluster private-ingress in resource group private-ingress-cl5k8g5wcstk, error: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-cl5k8g5wcstk" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20260630FromParam for cluster private-ingress in resource group private-ingress-cl5k8g5wcstk, error: context deadline exceeded
    ...
occurred
DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded
05:29:11 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc00177ecc0>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="pullsecret-test-7k8vgxjbrndr", caused by: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-7k8vgxjbrndr" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster pullsecret-hcp-cluster in resource group pullsecret-test-7k8vgxjbrndr, error: context deadline exceeded, error: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-7k8vgxjbrndr" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster pullsecret-hcp-cluster in resource group pullsecret-test-7k8vgxjbrndr, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc00177ecc0>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="pullsecret-test-7k8vgxjbrndr", caused by: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-7k8vgxjbrndr" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster pullsecret-hcp-cluster in resource group pullsecret-test-7k8vgxjbrndr, error: context deadline exceeded, error: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-7k8vgxjbrndr" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster pullsecret-hcp-cluster in resource group pullsecret-test-7k8vgxjbrndr, error: context deadline exceeded
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
04:55:12 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc000e46720>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-mj9b89zp6qrs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/764a11ed-2f54-474f-bdce-7ac64c3a7a20
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/764a11ed-2f54-474f-bdce-7ac64c3a7a20",
      "name": "764a11ed-2f54-474f-bdce-7ac64c3a7a20",
      "status": "Failed",
      "startTime": "2026-08-19T05:36:20.303691403Z",
      "endTime": "2026-08-19T05:55:29.476993829Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc000e46720>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-mj9b89zp6qrs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/764a11ed-2f54-474f-bdce-7ac64c3a7a20
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/764a11ed-2f54-474f-bdce-7ac64c3a7a20",
      "name": "764a11ed-2f54-474f-bdce-7ac64c3a7a20",
      "status": "Failed",
      "startTime": "2026-08-19T05:36:20.303691403Z",
      "endTime": "2026-08-19T05:55:29.476993829Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 4
e2e (4)
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
02:24:31 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc0010f8d20>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-2b48bcg6lskt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b7bfce51-c30e-43d1-a344-bf773679a5bf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b7bfce51-c30e-43d1-a344-bf773679a5bf",
      "name": "b7bfce51-c30e-43d1-a344-bf773679a5bf",
      "status": "Failed",
      "startTime": "2026-08-19T02:43:28.825953908Z",
      "endTime": "2026-08-19T03:02:32.100093145Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc0010f8d20>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-2b48bcg6lskt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b7bfce51-c30e-43d1-a344-bf773679a5bf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b7bfce51-c30e-43d1-a344-bf773679a5bf",
      "name": "b7bfce51-c30e-43d1-a344-bf773679a5bf",
      "status": "Failed",
      "startTime": "2026-08-19T02:43:28.825953908Z",
      "endTime": "2026-08-19T03:02:32.100093145Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; provider Microsoft.RedHatOpenShift
02:24:31 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260630preview and verify the ingress is internal · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000e87320>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-llwvfkrlbl4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c70527cf-5589-4a4f-a61e-87d227a6cadb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c70527cf-5589-4a4f-a61e-87d227a6cadb",
      "name": "c70527cf-5589-4a4f-a61e-87d227a6cadb",
      "status": "Failed",
      "startTime": "2026-08-19T02:41:07.832237891Z",
      "endTime": "2026-08-19T03:00:11.0783774Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000e87320>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-llwvfkrlbl4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c70527cf-5589-4a4f-a61e-87d227a6cadb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c70527cf-5589-4a4f-a61e-87d227a6cadb",
      "name": "c70527cf-5589-4a4f-a61e-87d227a6cadb",
      "status": "Failed",
      "startTime": "2026-08-19T02:41:07.832237891Z",
      "endTime": "2026-08-19T03:00:11.0783774Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; provider Microsoft.RedHatOpenShift
02:24:31 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc00091a9c0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-nzcg89cnmszj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/99a00b91-6d0c-48a1-9fdd-2970d104a1e6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/99a00b91-6d0c-48a1-9fdd-2970d104a1e6",
      "name": "99a00b91-6d0c-48a1-9fdd-2970d104a1e6",
      "status": "Failed",
      "startTime": "2026-08-19T02:42:34.622468609Z",
      "endTime": "2026-08-19T03:01:41.644884055Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc00091a9c0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-nzcg89cnmszj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/99a00b91-6d0c-48a1-9fdd-2970d104a1e6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/99a00b91-6d0c-48a1-9fdd-2970d104a1e6",
      "name": "99a00b91-6d0c-48a1-9fdd-2970d104a1e6",
      "status": "Failed",
      "startTime": "2026-08-19T02:42:34.622468609Z",
      "endTime": "2026-08-19T03:01:41.644884055Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
02:24:31 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc0009aad60>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-jd8m687tp89z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9027bd70-63ce-45e8-b2bf-31031a91295e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9027bd70-63ce-45e8-b2bf-31031a91295e",
      "name": "9027bd70-63ce-45e8-b2bf-31031a91295e",
      "status": "Failed",
      "startTime": "2026-08-19T02:42:26.094866031Z",
      "endTime": "2026-08-19T03:01:31.760732971Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc0009aad60>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-jd8m687tp89z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9027bd70-63ce-45e8-b2bf-31031a91295e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9027bd70-63ce-45e8-b2bf-31031a91295e",
      "name": "9027bd70-63ce-45e8-b2bf-31031a91295e",
      "status": "Failed",
      "startTime": "2026-08-19T02:42:26.094866031Z",
      "endTime": "2026-08-19T03:01:31.760732971Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 20
e2e (20)
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; provider Microsoft.RedHatOpenShift
02:00:06 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260630preview and verify the ingress is internal · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc00095a800>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-rbshm8pdpk8l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cfcd77a1-f130-4928-9266-77e6d35e2ca9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cfcd77a1-f130-4928-9266-77e6d35e2ca9",
      "name": "cfcd77a1-f130-4928-9266-77e6d35e2ca9",
      "status": "Failed",
      "startTime": "2026-08-19T02:02:06.555031993Z",
      "endTime": "2026-08-19T02:21:09.996463664Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc00095a800>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-rbshm8pdpk8l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cfcd77a1-f130-4928-9266-77e6d35e2ca9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cfcd77a1-f130-4928-9266-77e6d35e2ca9",
      "name": "cfcd77a1-f130-4928-9266-77e6d35e2ca9",
      "status": "Failed",
      "startTime": "2026-08-19T02:02:06.555031993Z",
      "endTime": "2026-08-19T02:21:09.996463664Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift
02:00:06 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-4-22-v2l547ms6nvl/cluster-candidate-4-22-4jlml7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f8d680>: 
    failed to create HCP cluster cluster-candidate-4-22-4jlml7: failed waiting for cluster="cluster-candidate-4-22-4jlml7" in resourcegroup="rg-candidate-4-22-v2l547ms6nvl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/37ac7da0-f61c-41aa-a8eb-17608806f235
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/37ac7da0-f61c-41aa-a8eb-17608806f235",
      "name": "37ac7da0-f61c-41aa-a8eb-17608806f235",
      "status": "Failed",
      "startTime": "2026-08-19T02:37:12.832748455Z",
      "endTime": "2026-08-19T02:56:20.716293366Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-4-22-v2l547ms6nvl/cluster-candidate-4-22-4jlml7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f8d680>: 
    failed to create HCP cluster cluster-candidate-4-22-4jlml7: failed waiting for cluster="cluster-candidate-4-22-4jlml7" in resourcegroup="rg-candidate-4-22-v2l547ms6nvl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/37ac7da0-f61c-41aa-a8eb-17608806f235
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/37ac7da0-f61c-41aa-a8eb-17608806f235",
      "name": "37ac7da0-f61c-41aa-a8eb-17608806f235",
      "status": "Failed",
      "startTime": "2026-08-19T02:37:12.832748455Z",
      "endTime": "2026-08-19T02:56:20.716293366Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
tls: failed to verify certificate: x509: certificate signed by unknown authority
02:00:06 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:175]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc000d24198>: 
    VerifySimpleWebApp failed: [strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-xjt5s.apps.aro.cilium-cl.j5nd.uksouth.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:175]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc000d24198>: 
    VerifySimpleWebApp failed: [strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-xjt5s.apps.aro.cilium-cl.j5nd.uksouth.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
02:00:06 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000e9cba0>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-kqnp9z8cmzh5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a6d2ca59-7ab4-4ed6-a90c-ef7ae4beb3b3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a6d2ca59-7ab4-4ed6-a90c-ef7ae4beb3b3",
      "name": "a6d2ca59-7ab4-4ed6-a90c-ef7ae4beb3b3",
      "status": "Failed",
      "startTime": "2026-08-19T02:49:58.123038157Z",
      "endTime": "2026-08-19T03:09:02.342492491Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000e9cba0>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-kqnp9z8cmzh5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a6d2ca59-7ab4-4ed6-a90c-ef7ae4beb3b3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a6d2ca59-7ab4-4ed6-a90c-ef7ae4beb3b3",
      "name": "a6d2ca59-7ab4-4ed6-a90c-ef7ae4beb3b3",
      "status": "Failed",
      "startTime": "2026-08-19T02:49:58.123038157Z",
      "endTime": "2026-08-19T03:09:02.342492491Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift
02:00:06 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-4-20-fkndtwz8x29s/cluster-candidate-4-20-kpswjg should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c0ef80>: 
    failed to create HCP cluster cluster-candidate-4-20-kpswjg: failed waiting for cluster="cluster-candidate-4-20-kpswjg" in resourcegroup="rg-candidate-4-20-fkndtwz8x29s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5ce8c6d6-e8b9-4d76-97a6-d8ba04a44b30
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5ce8c6d6-e8b9-4d76-97a6-d8ba04a44b30",
      "name": "5ce8c6d6-e8b9-4d76-97a6-d8ba04a44b30",
      "status": "Failed",
      "startTime": "2026-08-19T02:36:44.929925762Z",
      "endTime": "2026-08-19T02:55:50.765263478Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-4-20-fkndtwz8x29s/cluster-candidate-4-20-kpswjg should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c0ef80>: 
    failed to create HCP cluster cluster-candidate-4-20-kpswjg: failed waiting for cluster="cluster-candidate-4-20-kpswjg" in resourcegroup="rg-candidate-4-20-fkndtwz8x29s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5ce8c6d6-e8b9-4d76-97a6-d8ba04a44b30
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5ce8c6d6-e8b9-4d76-97a6-d8ba04a44b30",
      "name": "5ce8c6d6-e8b9-4d76-97a6-d8ba04a44b30",
      "status": "Failed",
      "startTime": "2026-08-19T02:36:44.929925762Z",
      "endTime": "2026-08-19T02:55:50.765263478Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
02:00:06 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000eca440>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-rq6k659hlc2b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/628698d5-0d72-472e-8dc8-dfdac91dfb4b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/628698d5-0d72-472e-8dc8-dfdac91dfb4b",
      "name": "628698d5-0d72-472e-8dc8-dfdac91dfb4b",
      "status": "Failed",
      "startTime": "2026-08-19T02:02:08.630685617Z",
      "endTime": "2026-08-19T02:21:09.934960233Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000eca440>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-rq6k659hlc2b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/628698d5-0d72-472e-8dc8-dfdac91dfb4b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/628698d5-0d72-472e-8dc8-dfdac91dfb4b",
      "name": "628698d5-0d72-472e-8dc8-dfdac91dfb4b",
      "status": "Failed",
      "startTime": "2026-08-19T02:02:08.630685617Z",
      "endTime": "2026-08-19T02:21:09.934960233Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
02:00:06 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-22-mq2h22" with version 4.22.8 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc00061d3a0>: 
    failed to create HCP cluster cluster-zstream-4-22-mq2h22: failed waiting for cluster="cluster-zstream-4-22-mq2h22" in resourcegroup="rg-zstream-upgrade-4-22-xtkph7svg6hh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bdb3b297-4dca-4d1a-b0e6-de304d1049b7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bdb3b297-4dca-4d1a-b0e6-de304d1049b7",
      "name": "bdb3b297-4dca-4d1a-b0e6-de304d1049b7",
      "status": "Failed",
      "startTime": "2026-08-19T02:40:32.481428416Z",
      "endTime": "2026-08-19T02:59:41.29757117Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-22-mq2h22" with version 4.22.8 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc00061d3a0>: 
    failed to create HCP cluster cluster-zstream-4-22-mq2h22: failed waiting for cluster="cluster-zstream-4-22-mq2h22" in resourcegroup="rg-zstream-upgrade-4-22-xtkph7svg6hh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bdb3b297-4dca-4d1a-b0e6-de304d1049b7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bdb3b297-4dca-4d1a-b0e6-de304d1049b7",
      "name": "bdb3b297-4dca-4d1a-b0e6-de304d1049b7",
      "status": "Failed",
      "startTime": "2026-08-19T02:40:32.481428416Z",
      "endTime": "2026-08-19T02:59:41.29757117Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
02:00:06 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-21-ln8w7f" with version 4.21.28 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc00013f6e0>: 
    failed to create HCP cluster cluster-zstream-4-21-ln8w7f: failed waiting for cluster="cluster-zstream-4-21-ln8w7f" in resourcegroup="rg-zstream-upgrade-4-21-tcfplz58xhnf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eb03c6ac-2187-4d92-bc26-df2c48cc7087
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eb03c6ac-2187-4d92-bc26-df2c48cc7087",
      "name": "eb03c6ac-2187-4d92-bc26-df2c48cc7087",
      "status": "Failed",
      "startTime": "2026-08-19T02:40:21.994801726Z",
      "endTime": "2026-08-19T02:59:21.262101435Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-21-ln8w7f" with version 4.21.28 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc00013f6e0>: 
    failed to create HCP cluster cluster-zstream-4-21-ln8w7f: failed waiting for cluster="cluster-zstream-4-21-ln8w7f" in resourcegroup="rg-zstream-upgrade-4-21-tcfplz58xhnf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eb03c6ac-2187-4d92-bc26-df2c48cc7087
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eb03c6ac-2187-4d92-bc26-df2c48cc7087",
      "name": "eb03c6ac-2187-4d92-bc26-df2c48cc7087",
      "status": "Failed",
      "startTime": "2026-08-19T02:40:21.994801726Z",
      "endTime": "2026-08-19T02:59:21.262101435Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
02:00:06 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:98]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc000a9ab40>: 
    failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-7wkzpgtrr9pp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/32f6d177-e108-4c1d-b722-a68f242f9033
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/32f6d177-e108-4c1d-b722-a68f242f9033",
      "name": "32f6d177-e108-4c1d-b722-a68f242f9033",
      "status": "Failed",
      "startTime": "2026-08-19T02:54:06.939102032Z",
      "endTime": "2026-08-19T03:13:12.339453832Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:98]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc000a9ab40>: 
    failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-7wkzpgtrr9pp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/32f6d177-e108-4c1d-b722-a68f242f9033
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/32f6d177-e108-4c1d-b722-a68f242f9033",
      "name": "32f6d177-e108-4c1d-b722-a68f242f9033",
      "status": "Failed",
      "startTime": "2026-08-19T02:54:06.939102032Z",
      "endTime": "2026-08-19T03:13:12.339453832Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
02:00:06 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:182]: failed to create HCP cluster "cp-ystream-upgrade-4-22-d97nxs" with install version 4.21.28
Unexpected error:
    <*fmt.wrapError | 0xc00087d500>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-d97nxs: failed waiting for cluster="cp-ystream-upgrade-4-22-d97nxs" in resourcegroup="rg-cp-ystream-upgrade-4-22-5lw87j2jhmv8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/77d1c726-ebe5-42a5-b8e1-086ef2e9bc02
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/77d1c726-ebe5-42a5-b8e1-086ef2e9bc02",
      "name": "77d1c726-ebe5-42a5-b8e1-086ef2e9bc02",
      "status": "Failed",
      "startTime": "2026-08-19T02:41:27.387753378Z",
      "endTime": "2026-08-19T03:00:31.886645584Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:182]: failed to create HCP cluster "cp-ystream-upgrade-4-22-d97nxs" with install version 4.21.28
Unexpected error:
    <*fmt.wrapError | 0xc00087d500>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-d97nxs: failed waiting for cluster="cp-ystream-upgrade-4-22-d97nxs" in resourcegroup="rg-cp-ystream-upgrade-4-22-5lw87j2jhmv8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/77d1c726-ebe5-42a5-b8e1-086ef2e9bc02
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/77d1c726-ebe5-42a5-b8e1-086ef2e9bc02",
      "name": "77d1c726-ebe5-42a5-b8e1-086ef2e9bc02",
      "status": "Failed",
      "startTime": "2026-08-19T02:41:27.387753378Z",
      "endTime": "2026-08-19T03:00:31.886645584Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift
02:00:06 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000923f40>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-886kmmqnwq8r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a873ac5e-6930-421b-80c5-b0073727d292
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a873ac5e-6930-421b-80c5-b0073727d292",
      "name": "a873ac5e-6930-421b-80c5-b0073727d292",
      "status": "Failed",
      "startTime": "2026-08-19T02:03:10.761261438Z",
      "endTime": "2026-08-19T02:22:19.897779073Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000923f40>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-886kmmqnwq8r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a873ac5e-6930-421b-80c5-b0073727d292
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a873ac5e-6930-421b-80c5-b0073727d292",
      "name": "a873ac5e-6930-421b-80c5-b0073727d292",
      "status": "Failed",
      "startTime": "2026-08-19T02:03:10.761261438Z",
      "endTime": "2026-08-19T02:22:19.897779073Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
02:00:06 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:138]: failed to wait for second cluster "basic-hcp-cluster2" to complete creation (timeout '20.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc000b74600>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/603abdb0-e1da-4fca-b629-437a70a04359
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/603abdb0-e1da-4fca-b629-437a70a04359",
      "name": "603abdb0-e1da-4fca-b629-437a70a04359",
      "status": "Failed",
      "startTime": "2026-08-19T02:03:09.728610324Z",
      "endTime": "2026-08-19T02:22:09.779874488Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:138]: failed to wait for second cluster "basic-hcp-cluster2" to complete creation (timeout '20.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc000b74600>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/603abdb0-e1da-4fca-b629-437a70a04359
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/603abdb0-e1da-4fca-b629-437a70a04359",
      "name": "603abdb0-e1da-4fca-b629-437a70a04359",
      "status": "Failed",
      "startTime": "2026-08-19T02:03:09.728610324Z",
      "endTime": "2026-08-19T02:22:09.779874488Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
02:00:06 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc0009cec40>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-wr59bczt45g4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/03f1c6ac-7d98-4fee-9bc6-a6d3cbb8f4bb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/03f1c6ac-7d98-4fee-9bc6-a6d3cbb8f4bb",
      "name": "03f1c6ac-7d98-4fee-9bc6-a6d3cbb8f4bb",
      "status": "Failed",
      "startTime": "2026-08-19T02:50:32.97213323Z",
      "endTime": "2026-08-19T03:09:32.109624172Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc0009cec40>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-wr59bczt45g4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/03f1c6ac-7d98-4fee-9bc6-a6d3cbb8f4bb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/03f1c6ac-7d98-4fee-9bc6-a6d3cbb8f4bb",
      "name": "03f1c6ac-7d98-4fee-9bc6-a6d3cbb8f4bb",
      "status": "Failed",
      "startTime": "2026-08-19T02:50:32.97213323Z",
      "endTime": "2026-08-19T03:09:32.109624172Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift
02:00:06 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-5-0-zjc9p9k8fp8k/cluster-candidate-5-0-kz87c7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f9e720>: 
    failed to create HCP cluster cluster-candidate-5-0-kz87c7: failed waiting for cluster="cluster-candidate-5-0-kz87c7" in resourcegroup="rg-candidate-5-0-zjc9p9k8fp8k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a3081846-fb1d-49f7-9bc5-d276f1e7c8f7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a3081846-fb1d-49f7-9bc5-d276f1e7c8f7",
      "name": "a3081846-fb1d-49f7-9bc5-d276f1e7c8f7",
      "status": "Failed",
      "startTime": "2026-08-19T02:38:11.46423025Z",
      "endTime": "2026-08-19T02:57:21.467415333Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-5-0-zjc9p9k8fp8k/cluster-candidate-5-0-kz87c7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f9e720>: 
    failed to create HCP cluster cluster-candidate-5-0-kz87c7: failed waiting for cluster="cluster-candidate-5-0-kz87c7" in resourcegroup="rg-candidate-5-0-zjc9p9k8fp8k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a3081846-fb1d-49f7-9bc5-d276f1e7c8f7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a3081846-fb1d-49f7-9bc5-d276f1e7c8f7",
      "name": "a3081846-fb1d-49f7-9bc5-d276f1e7c8f7",
      "status": "Failed",
      "startTime": "2026-08-19T02:38:11.46423025Z",
      "endTime": "2026-08-19T02:57:21.467415333Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
02:00:06 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc00091cd80>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-45gzclqzvpj8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e88bd8a9-38d6-4536-95d7-87ee61383c3d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e88bd8a9-38d6-4536-95d7-87ee61383c3d",
      "name": "e88bd8a9-38d6-4536-95d7-87ee61383c3d",
      "status": "Failed",
      "startTime": "2026-08-19T02:42:53.967643939Z",
      "endTime": "2026-08-19T03:02:02.350743485Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc00091cd80>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-45gzclqzvpj8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e88bd8a9-38d6-4536-95d7-87ee61383c3d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e88bd8a9-38d6-4536-95d7-87ee61383c3d",
      "name": "e88bd8a9-38d6-4536-95d7-87ee61383c3d",
      "status": "Failed",
      "startTime": "2026-08-19T02:42:53.967643939Z",
      "endTime": "2026-08-19T03:02:02.350743485Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; provider Microsoft.RedHatOpenShift
02:00:06 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000f7cbe0>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-55vcnkhwfhv9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/14cadb61-ba81-4ce7-bcb4-95b9e782e1b8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/14cadb61-ba81-4ce7-bcb4-95b9e782e1b8",
      "name": "14cadb61-ba81-4ce7-bcb4-95b9e782e1b8",
      "status": "Failed",
      "startTime": "2026-08-19T02:04:34.114580789Z",
      "endTime": "2026-08-19T02:23:39.974581196Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000f7cbe0>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-55vcnkhwfhv9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/14cadb61-ba81-4ce7-bcb4-95b9e782e1b8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/14cadb61-ba81-4ce7-bcb4-95b9e782e1b8",
      "name": "14cadb61-ba81-4ce7-bcb4-95b9e782e1b8",
      "status": "Failed",
      "startTime": "2026-08-19T02:04:34.114580789Z",
      "endTime": "2026-08-19T02:23:39.974581196Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; provider Microsoft.RedHatOpenShift
02:00:06 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-4-21-8ckjgqwvdkqt/cluster-candidate-4-21-5vg74m should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f9e160>: 
    failed to create HCP cluster cluster-candidate-4-21-5vg74m: failed waiting for cluster="cluster-candidate-4-21-5vg74m" in resourcegroup="rg-candidate-4-21-8ckjgqwvdkqt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cdcbadeb-fba6-4cc0-90c7-9d1c93346bb9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cdcbadeb-fba6-4cc0-90c7-9d1c93346bb9",
      "name": "cdcbadeb-fba6-4cc0-90c7-9d1c93346bb9",
      "status": "Failed",
      "startTime": "2026-08-19T02:37:11.353112442Z",
      "endTime": "2026-08-19T02:56:20.740956603Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-4-21-8ckjgqwvdkqt/cluster-candidate-4-21-5vg74m should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f9e160>: 
    failed to create HCP cluster cluster-candidate-4-21-5vg74m: failed waiting for cluster="cluster-candidate-4-21-5vg74m" in resourcegroup="rg-candidate-4-21-8ckjgqwvdkqt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cdcbadeb-fba6-4cc0-90c7-9d1c93346bb9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cdcbadeb-fba6-4cc0-90c7-9d1c93346bb9",
      "name": "cdcbadeb-fba6-4cc0-90c7-9d1c93346bb9",
      "status": "Failed",
      "startTime": "2026-08-19T02:37:11.353112442Z",
      "endTime": "2026-08-19T02:56:20.740956603Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift
02:00:06 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000f00ec0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-6k6scsrzs89w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/002cf859-3aea-4235-ade3-bfe0bd472945
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/002cf859-3aea-4235-ade3-bfe0bd472945",
      "name": "002cf859-3aea-4235-ade3-bfe0bd472945",
      "status": "Failed",
      "startTime": "2026-08-19T02:46:18.55837028Z",
      "endTime": "2026-08-19T03:05:22.599243963Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000f00ec0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-6k6scsrzs89w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/002cf859-3aea-4235-ade3-bfe0bd472945
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/002cf859-3aea-4235-ade3-bfe0bd472945",
      "name": "002cf859-3aea-4235-ade3-bfe0bd472945",
      "status": "Failed",
      "startTime": "2026-08-19T02:46:18.55837028Z",
      "endTime": "2026-08-19T03:05:22.599243963Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift
02:00:06 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc0008849a0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-wql9556ddm97" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d68f2c54-5fd0-4ba4-a58c-3b05ce6cf2ec
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d68f2c54-5fd0-4ba4-a58c-3b05ce6cf2ec",
      "name": "d68f2c54-5fd0-4ba4-a58c-3b05ce6cf2ec",
      "status": "Failed",
      "startTime": "2026-08-19T02:03:39.569370982Z",
      "endTime": "2026-08-19T02:22:39.7189338Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc0008849a0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-wql9556ddm97" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d68f2c54-5fd0-4ba4-a58c-3b05ce6cf2ec
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d68f2c54-5fd0-4ba4-a58c-3b05ce6cf2ec",
      "name": "d68f2c54-5fd0-4ba4-a58c-3b05ce6cf2ec",
      "status": "Failed",
      "startTime": "2026-08-19T02:03:39.569370982Z",
      "endTime": "2026-08-19T02:22:39.7189338Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
02:00:06 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-20-zhbw8r" with version 4.20.33 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000e626a0>: 
    failed to create HCP cluster cluster-zstream-4-20-zhbw8r: failed waiting for cluster="cluster-zstream-4-20-zhbw8r" in resourcegroup="rg-zstream-upgrade-4-20-ksfz2k8984rm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f3da66f1-7784-4899-91fd-4bc2f63735e3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f3da66f1-7784-4899-91fd-4bc2f63735e3",
      "name": "f3da66f1-7784-4899-91fd-4bc2f63735e3",
      "status": "Failed",
      "startTime": "2026-08-19T02:39:49.537962227Z",
      "endTime": "2026-08-19T02:58:51.273135808Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-20-zhbw8r" with version 4.20.33 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000e626a0>: 
    failed to create HCP cluster cluster-zstream-4-20-zhbw8r: failed waiting for cluster="cluster-zstream-4-20-zhbw8r" in resourcegroup="rg-zstream-upgrade-4-20-ksfz2k8984rm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f3da66f1-7784-4899-91fd-4bc2f63735e3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f3da66f1-7784-4899-91fd-4bc2f63735e3",
      "name": "f3da66f1-7784-4899-91fd-4bc2f63735e3",
      "status": "Failed",
      "startTime": "2026-08-19T02:39:49.537962227Z",
      "endTime": "2026-08-19T02:58:51.273135808Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred