CIHealth
<
Single Day: Oct 22▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: PROD

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
30m 0s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a unknown failed n/a 1 non-artifact-backed failure (no junit artifacts)
1h 47m 20s periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly
post-goodmerged PR
No additional run metadata captured
 canadacentral e2e failed n/a 10
e2e (10)
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
23:00:29 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x8dce7251b0>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x8dce7251b0>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
23:00:29 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0xe4d0c558c40>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0xe4d0c558c40>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0
23:00:29 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-22-5j4sbg"
Expected success, but got an error:
    <*fmt.wrapError | 0x26e6691a0320>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.003s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.003s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0x26e6691a0300>{
            observation: <*errors.errorString | 0x26e66a200e10>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0x26e6691a02c0>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-22-5j4sbg"
Expected success, but got an error:
    <*fmt.wrapError | 0x26e6691a0320>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.003s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.003s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0x26e6691a0300>{
            observation: <*errors.errorString | 0x26e66a200e10>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0x26e6691a02c0>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0
23:00:29 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-5-0-w29x8s"
Expected success, but got an error:
    <*fmt.wrapError | 0x2369de188ea0>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0x2369de188e60>{
            observation: <*errors.errorString | 0x2369e043f830>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0x2369de188d80>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-5-0-w29x8s"
Expected success, but got an error:
    <*fmt.wrapError | 0x2369de188ea0>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0x2369de188e60>{
            observation: <*errors.errorString | 0x2369e043f830>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0x2369de188d80>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
23:00:29 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x663da188400>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x663da188400>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
23:00:29 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x14719e7fa150>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x14719e7fa150>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0
23:00:29 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-21-zn6lkv"
Expected success, but got an error:
    <*fmt.wrapError | 0x1288c746e940>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
        err: <*fmt.wrapError | 0x1288c746e900>{
            msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
            err: <*fmt.wrapError | 0x1288c746e8c0>{
                msg: "client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
                err: <*errors.errorString | 0x1288c8648ef0>{
                    s: "rate: Wait(n=1) would exceed context deadline",
                },
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-21-zn6lkv"
Expected success, but got an error:
    <*fmt.wrapError | 0x1288c746e940>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
        err: <*fmt.wrapError | 0x1288c746e900>{
            msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
            err: <*fmt.wrapError | 0x1288c746e8c0>{
                msg: "client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
                err: <*errors.errorString | 0x1288c8648ef0>{
                    s: "rate: Wait(n=1) would exceed context deadline",
                },
            },
        },
    }
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
23:00:29 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x37eb40681150>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x37eb40681150>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0
23:00:29 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-20-z2m2qz"
Expected success, but got an error:
    <*fmt.wrapError | 0x5840fb59bc0>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0x5840fb59b40>{
            observation: <*errors.errorString | 0x584102eb3b0>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0x5840fb59ac0>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-20-z2m2qz"
Expected success, but got an error:
    <*fmt.wrapError | 0x5840fb59bc0>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0x5840fb59b40>{
            observation: <*errors.errorString | 0x584102eb3b0>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0x5840fb59ac0>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0
23:00:29 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-23-9frg2v"
Expected success, but got an error:
    <*fmt.wrapError | 0x26bc41040e40>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0x26bc41040dc0>{
            observation: <*errors.errorString | 0x26bc40ab97d0>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0x26bc41040d80>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-23-9frg2v"
Expected success, but got an error:
    <*fmt.wrapError | 0x26bc41040e40>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0x26bc41040dc0>{
            observation: <*errors.errorString | 0x26bc40ab97d0>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0x26bc41040d80>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
30m 2s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a unknown failed n/a 1 non-artifact-backed failure (no junit artifacts)
5h 55m 17s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 uksouth e2e failed n/a 15
e2e (15)
<Azure error internals omitted>
14:20:40 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0x15ce09908640>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-lz7wggbwgh5c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/30507e2e-a247-48d9-a1d3-5e74fda3481f
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/30507e2e-a247-48d9-a1d3-5e74fda3481f
            correlation ID: bd254d01-316e-4e34-af7b-a66cb2bc888c
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0x15ce09908640>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-lz7wggbwgh5c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/30507e2e-a247-48d9-a1d3-5e74fda3481f
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/30507e2e-a247-48d9-a1d3-5e74fda3481f
            correlation ID: bd254d01-316e-4e34-af7b-a66cb2bc888c
            client request ID: unavailable
occurred
<Azure error internals omitted>
14:20:40 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:175]: failed to get admin REST config for cluster np-version-upgrade-cluster-6klfkt
Unexpected error:
    <*fmt.wrapError | 0x31e04d7a70c0>: 
    failed waiting for hcpCluster="np-version-upgrade-cluster-6klfkt" in resourcegroup="rg-np-version-upgrade-6klfkt-x72h4v22zqrn" to finish getting creds: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0b189d0f-2181-4aed-9e7f-51e3cb6c01ee
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0b189d0f-2181-4aed-9e7f-51e3cb6c01ee
            correlation ID: acc4317f-332d-41ad-92b6-3b3da8362c30
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:175]: failed to get admin REST config for cluster np-version-upgrade-cluster-6klfkt
Unexpected error:
    <*fmt.wrapError | 0x31e04d7a70c0>: 
    failed waiting for hcpCluster="np-version-upgrade-cluster-6klfkt" in resourcegroup="rg-np-version-upgrade-6klfkt-x72h4v22zqrn" to finish getting creds: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0b189d0f-2181-4aed-9e7f-51e3cb6c01ee
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0b189d0f-2181-4aed-9e7f-51e3cb6c01ee
            correlation ID: acc4317f-332d-41ad-92b6-3b3da8362c30
            client request ID: unavailable
occurred
failed waiting for nodepool="<nodepool>" in cluster="<cluster>" resourcegroup="<resource-group>" to finish updating: GET <url>
14:20:40 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapError | 0x32f3fb206560>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-qk6fnz9248pm" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8a81c298-fb2e-4c06-ab5b-f047a74ffe2b
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8a81c298-fb2e-4c06-ab5b-f047a74ffe2b
            correlation ID: bd21a41c-3537-483b-a104-ebc5b06bde9b
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapError | 0x32f3fb206560>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-qk6fnz9248pm" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8a81c298-fb2e-4c06-ab5b-f047a74ffe2b
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8a81c298-fb2e-4c06-ab5b-f047a74ffe2b
            correlation ID: bd21a41c-3537-483b-a104-ebc5b06bde9b
            client request ID: unavailable
occurred
<Azure error internals omitted>
14:20:40 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x1745c0262280>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-ddrtzn7r9n6f" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aa390de3-b57c-4004-ae00-8d256a07e3fb
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aa390de3-b57c-4004-ae00-8d256a07e3fb
            correlation ID: 23755ba0-9dc6-42ab-bdd9-3208cee1312c
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x1745c0262280>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-ddrtzn7r9n6f" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aa390de3-b57c-4004-ae00-8d256a07e3fb
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aa390de3-b57c-4004-ae00-8d256a07e3fb
            correlation ID: 23755ba0-9dc6-42ab-bdd9-3208cee1312c
            client request ID: unavailable
occurred
failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
14:20:40 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:600]: failed to create nodepool npdg-4-21
Unexpected error:
    <*fmt.wrapError | 0x6fc68b1db40>: 
    failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-xkft46" in resourcegroup="rg-np-downgrade-xkft46-4q28hv2srddf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fb2f9bf6-57ed-44ee-a29f-f7d739da032d
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fb2f9bf6-57ed-44ee-a29f-f7d739da032d
            correlation ID: f554d9ad-22a3-4412-a6b0-c83016dc310e
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:600]: failed to create nodepool npdg-4-21
Unexpected error:
    <*fmt.wrapError | 0x6fc68b1db40>: 
    failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-xkft46" in resourcegroup="rg-np-downgrade-xkft46-4q28hv2srddf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fb2f9bf6-57ed-44ee-a29f-f7d739da032d
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fb2f9bf6-57ed-44ee-a29f-f7d739da032d
            correlation ID: f554d9ad-22a3-4412-a6b0-c83016dc310e
            client request ID: unavailable
occurred
<Azure error internals omitted>
14:20:40 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x32d7aaca6180>: 
    resource group "nodepool-delete-cd2q42rnhv9p": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-cd2q42rnhv9p" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8abc07fd-a04c-472f-9209-d29535531a3b
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8abc07fd-a04c-472f-9209-d29535531a3b
            correlation ID: 270a5375-3ee5-4792-82aa-8de0a5f92543
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x32d7aaca6180>: 
    resource group "nodepool-delete-cd2q42rnhv9p": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-cd2q42rnhv9p" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8abc07fd-a04c-472f-9209-d29535531a3b
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8abc07fd-a04c-472f-9209-d29535531a3b
            correlation ID: 270a5375-3ee5-4792-82aa-8de0a5f92543
            client request ID: unavailable
occurred
<Azure error internals omitted>
14:20:40 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:175]: failed to get admin REST config for cluster np-version-upgrade-cluster-rt5grp
Unexpected error:
    <*fmt.wrapError | 0x248e5dd5cbc0>: 
    failed waiting for hcpCluster="np-version-upgrade-cluster-rt5grp" in resourcegroup="rg-np-version-upgrade-rt5grp-4q86fvkpwgjw" to finish getting creds: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5874d223-8094-4a83-b531-392c43d65224
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5874d223-8094-4a83-b531-392c43d65224
            correlation ID: 2733429a-1831-4753-b8fd-9cd662863c39
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:175]: failed to get admin REST config for cluster np-version-upgrade-cluster-rt5grp
Unexpected error:
    <*fmt.wrapError | 0x248e5dd5cbc0>: 
    failed waiting for hcpCluster="np-version-upgrade-cluster-rt5grp" in resourcegroup="rg-np-version-upgrade-rt5grp-4q86fvkpwgjw" to finish getting creds: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5874d223-8094-4a83-b531-392c43d65224
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5874d223-8094-4a83-b531-392c43d65224
            correlation ID: 2733429a-1831-4753-b8fd-9cd662863c39
            client request ID: unavailable
occurred
failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
14:20:40 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
    <*fmt.wrapError | 0x11adfdef64a0>: 
    failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-nm2jdjsx4m9t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7ab31f32-e5a9-4966-9e78-a32d1786c989
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7ab31f32-e5a9-4966-9e78-a32d1786c989
            correlation ID: f811a401-7f9a-42bd-861f-59e24ffc103f
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
    <*fmt.wrapError | 0x11adfdef64a0>: 
    failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-nm2jdjsx4m9t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7ab31f32-e5a9-4966-9e78-a32d1786c989
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7ab31f32-e5a9-4966-9e78-a32d1786c989
            correlation ID: f811a401-7f9a-42bd-861f-59e24ffc103f
            client request ID: unavailable
occurred
failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
14:20:40 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.42
Unexpected error:
    <*fmt.wrapError | 0x42a73e640c0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-cqs6bm" in resourcegroup="rg-np-version-upgrade-cqs6bm-bdk46hm6gz2t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a7be4883-49a5-4223-a7fd-b85c36625889
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a7be4883-49a5-4223-a7fd-b85c36625889
            correlation ID: e1f797c4-9d76-4caf-85e3-d444fb84110b
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.42
Unexpected error:
    <*fmt.wrapError | 0x42a73e640c0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-cqs6bm" in resourcegroup="rg-np-version-upgrade-cqs6bm-bdk46hm6gz2t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a7be4883-49a5-4223-a7fd-b85c36625889
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a7be4883-49a5-4223-a7fd-b85c36625889
            correlation ID: e1f797c4-9d76-4caf-85e3-d444fb84110b
            client request ID: unavailable
occurred
failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
14:20:40 UTC · failed at e2e · test Nodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:143]: failed to create nodepool des-np with DES
Unexpected error:
    <*fmt.wrapError | 0x1628bc606300>: 
    failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-ngqf2fwd7rsk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e1fddbdc-ff80-4316-bce0-927ca36e9342
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e1fddbdc-ff80-4316-bce0-927ca36e9342
            correlation ID: 6c7f6da2-0648-48d3-a365-326009a54a6d
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:143]: failed to create nodepool des-np with DES
Unexpected error:
    <*fmt.wrapError | 0x1628bc606300>: 
    failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-ngqf2fwd7rsk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e1fddbdc-ff80-4316-bce0-927ca36e9342
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e1fddbdc-ff80-4316-bce0-927ca36e9342
            correlation ID: 6c7f6da2-0648-48d3-a365-326009a54a6d
            client request ID: unavailable
occurred
failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
14:20:40 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.42
Unexpected error:
    <*fmt.wrapError | 0x8e881283980>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-8tlm65" in resourcegroup="rg-np-noedge-8tlm65-6rtrf6fxs8pp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/657d92b1-0420-465c-a20f-051929954d68
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/657d92b1-0420-465c-a20f-051929954d68
            correlation ID: da25ea79-7ee4-4aa5-a3cd-24d74907d5d3
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.42
Unexpected error:
    <*fmt.wrapError | 0x8e881283980>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-8tlm65" in resourcegroup="rg-np-noedge-8tlm65-6rtrf6fxs8pp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/657d92b1-0420-465c-a20f-051929954d68
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/657d92b1-0420-465c-a20f-051929954d68
            correlation ID: da25ea79-7ee4-4aa5-a3cd-24d74907d5d3
            client request ID: unavailable
occurred
<Azure error internals omitted>
14:20:40 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x2a26cf54c078>: 
    resource group "np-autoscaling-ncn4v65w96fx": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-ncn4v65w96fx" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d43b55c4-7331-4381-b4fd-71f64a776433
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d43b55c4-7331-4381-b4fd-71f64a776433
            correlation ID: 1d901265-7895-4b99-bf69-cdc848b0d0b5
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x2a26cf54c078>: 
    resource group "np-autoscaling-ncn4v65w96fx": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-ncn4v65w96fx" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d43b55c4-7331-4381-b4fd-71f64a776433
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d43b55c4-7331-4381-b4fd-71f64a776433
            correlation ID: 1d901265-7895-4b99-bf69-cdc848b0d0b5
            client request ID: unavailable
occurred
failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
14:20:40 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.43
Unexpected error:
    <*fmt.wrapError | 0x378ab2d1b4e0>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-p9x242" in resourcegroup="rg-np-skip-minor-p9x242-tqcdq45mhp7v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/773f88ac-95e8-4da7-bc09-57a818deb175
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/773f88ac-95e8-4da7-bc09-57a818deb175
            correlation ID: 8aa44ab4-58c2-423f-9a40-2f66225c45a7
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.43
Unexpected error:
    <*fmt.wrapError | 0x378ab2d1b4e0>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-p9x242" in resourcegroup="rg-np-skip-minor-p9x242-tqcdq45mhp7v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/773f88ac-95e8-4da7-bc09-57a818deb175
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/773f88ac-95e8-4da7-bc09-57a818deb175
            correlation ID: 8aa44ab4-58c2-423f-9a40-2f66225c45a7
            client request ID: unavailable
occurred
failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
14:20:40 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:740]: failed to create nodepool npdg-422-420
Unexpected error:
    <*fmt.wrapError | 0x23e7733bb0a0>: 
    failed to create NodePool npdg-422-420: failed waiting for nodepool="npdg-422-420" for cluster "np-dg-minor-q27q4m" in resourcegroup="rg-np-dg-minor-q27q4m-lfz4ljlvt6cw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d8c2dbec-e136-4d1f-8130-7b66c5f0a6c1
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d8c2dbec-e136-4d1f-8130-7b66c5f0a6c1
            correlation ID: 4b13a3a4-dc33-47f4-9b01-e6df32158296
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:740]: failed to create nodepool npdg-422-420
Unexpected error:
    <*fmt.wrapError | 0x23e7733bb0a0>: 
    failed to create NodePool npdg-422-420: failed waiting for nodepool="npdg-422-420" for cluster "np-dg-minor-q27q4m" in resourcegroup="rg-np-dg-minor-q27q4m-lfz4ljlvt6cw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d8c2dbec-e136-4d1f-8130-7b66c5f0a6c1
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d8c2dbec-e136-4d1f-8130-7b66c5f0a6c1
            correlation ID: 4b13a3a4-dc33-47f4-9b01-e6df32158296
            client request ID: unavailable
occurred
<Azure error internals omitted>
14:20:40 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x288e5c900060>: 
    resource group "ephemeral-osdisk-4gswgswxhhgn": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-4gswgswxhhgn" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e9d42f84-ef34-4bd9-97b4-3ce13c3377ef
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e9d42f84-ef34-4bd9-97b4-3ce13c3377ef
            correlation ID: c098aa4e-9c5f-48a5-ac6b-501930ee0285
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x288e5c900060>: 
    resource group "ephemeral-osdisk-4gswgswxhhgn": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-4gswgswxhhgn" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e9d42f84-ef34-4bd9-97b4-3ce13c3377ef
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE UNAVAILABLE
    --------------------------------------------------------------------------------
    Response contained no body
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e9d42f84-ef34-4bd9-97b4-3ce13c3377ef
            correlation ID: c098aa4e-9c5f-48a5-ac6b-501930ee0285
            client request ID: unavailable
occurred
2h 29m 47s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 westeurope n/a passed n/a 0 n/a
2h 29m 26s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 westeurope e2e failed n/a 2
e2e (2)
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: NotReady: * NodeHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?; provider Microsoft.RedHatOpenShift
04:22:40 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:177]: failed to create node pool "np-lp5s2l" for cluster "cluster-candidate-5-0-lp5s2l"
Unexpected error:
    <*fmt.wrapError | 0x3c5070b80fa0>: 
    failed to create NodePool np-lp5s2l: failed waiting for nodepool="np-lp5s2l" for cluster "cluster-candidate-5-0-lp5s2l" in resourcegroup="rg-candidate-5-0-m8vcgvcqk4pg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/fd2644a3-f725-418e-af3f-2985a0768c42
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/fd2644a3-f725-418e-af3f-2985a0768c42",
      "name": "fd2644a3-f725-418e-af3f-2985a0768c42",
      "status": "Failed",
      "startTime": "2026-10-22T05:20:50.362804863Z",
      "endTime": "2026-10-22T05:39:55.652634149Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u1j3l3a4s2x8k8u-np-lp5s2l-lng7p-xl2t7: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u1j3l3a4s2x8k8u-np-lp5s2l-lng7p-xl2t7: NotReady: * NodeHealthy:\n  * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/fd2644a3-f725-418e-af3f-2985a0768c42
            correlation ID: b2009fac-4672-460e-84fc-49211ba0f615
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:177]: failed to create node pool "np-lp5s2l" for cluster "cluster-candidate-5-0-lp5s2l"
Unexpected error:
    <*fmt.wrapError | 0x3c5070b80fa0>: 
    failed to create NodePool np-lp5s2l: failed waiting for nodepool="np-lp5s2l" for cluster "cluster-candidate-5-0-lp5s2l" in resourcegroup="rg-candidate-5-0-m8vcgvcqk4pg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/fd2644a3-f725-418e-af3f-2985a0768c42
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/fd2644a3-f725-418e-af3f-2985a0768c42",
      "name": "fd2644a3-f725-418e-af3f-2985a0768c42",
      "status": "Failed",
      "startTime": "2026-10-22T05:20:50.362804863Z",
      "endTime": "2026-10-22T05:39:55.652634149Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u1j3l3a4s2x8k8u-np-lp5s2l-lng7p-xl2t7: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u1j3l3a4s2x8k8u-np-lp5s2l-lng7p-xl2t7: NotReady: * NodeHealthy:\n  * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/fd2644a3-f725-418e-af3f-2985a0768c42
            correlation ID: b2009fac-4672-460e-84fc-49211ba0f615
            client request ID: unavailable
occurred
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
04:22:40 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x3eb0083dc060>: 
    resource group "private-keyvault-dtwhsct926n6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-dtwhsct926n6" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/baf15df3-25ce-45bb-9f9e-ce6a52637752
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/baf15df3-25ce-45bb-9f9e-ce6a52637752",
      "name": "baf15df3-25ce-45bb-9f9e-ce6a52637752",
      "status": "Failed",
      "startTime": "2026-10-22T05:08:07.811493592Z",
      "endTime": "2026-10-22T05:32:16.130563503Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tjkddd25goskik0nf1lqaqp6mgu12qg still exists (deletion dispatched at 2026-10-22T05:08:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/baf15df3-25ce-45bb-9f9e-ce6a52637752
            correlation ID: a461bf17-668e-42e4-a6f7-c14803c5d886
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x3eb0083dc060>: 
    resource group "private-keyvault-dtwhsct926n6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-dtwhsct926n6" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/baf15df3-25ce-45bb-9f9e-ce6a52637752
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/baf15df3-25ce-45bb-9f9e-ce6a52637752",
      "name": "baf15df3-25ce-45bb-9f9e-ce6a52637752",
      "status": "Failed",
      "startTime": "2026-10-22T05:08:07.811493592Z",
      "endTime": "2026-10-22T05:32:16.130563503Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tjkddd25goskik0nf1lqaqp6mgu12qg still exists (deletion dispatched at 2026-10-22T05:08:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/baf15df3-25ce-45bb-9f9e-ce6a52637752
            correlation ID: a461bf17-668e-42e4-a6f7-c14803c5d886
            client request ID: unavailable
occurred
2h 16m 35s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 brazilsouth n/a passed n/a 0 n/a
2h 16m 23s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 eastus2 n/a passed n/a 0 n/a
1h 54m 19s periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 australiaeast n/a passed n/a 0 n/a
2h 0m 28s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 centralindia n/a passed n/a 0 n/a
3h 7m 37s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 eastus2euap e2e failed n/a 62
e2e (62)
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should be able to create an HCP cluster then delete it by deleting the customer resource group · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:92]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0x2ecb44a0c500>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-s6r6htggh6z7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2fc31b58-fcb3-4fef-814b-d7ab54f186d7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2fc31b58-fcb3-4fef-814b-d7ab54f186d7",
      "name": "2fc31b58-fcb3-4fef-814b-d7ab54f186d7",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:05.760617049Z",
      "endTime": "2026-10-22T02:03:06.678299552Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2fc31b58-fcb3-4fef-814b-d7ab54f186d7
            correlation ID: bb4615a3-e6b5-4e4b-a502-f87305616869
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:92]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0x2ecb44a0c500>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-s6r6htggh6z7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2fc31b58-fcb3-4fef-814b-d7ab54f186d7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2fc31b58-fcb3-4fef-814b-d7ab54f186d7",
      "name": "2fc31b58-fcb3-4fef-814b-d7ab54f186d7",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:05.760617049Z",
      "endTime": "2026-10-22T02:03:06.678299552Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2fc31b58-fcb3-4fef-814b-d7ab54f186d7
            correlation ID: bb4615a3-e6b5-4e4b-a502-f87305616869
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [capi-provider, kube-apiserver]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-5-0-bzlw46" with version 5.0.0-rc.6 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x202d14170440>: 
    failed to create HCP cluster cluster-zstream-5-0-bzlw46: failed waiting for cluster="cluster-zstream-5-0-bzlw46" in resourcegroup="rg-zstream-upgrade-5-0-l5ckf5pqgjzq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b119ec20-94fb-4593-816b-4d56620a6799
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b119ec20-94fb-4593-816b-4d56620a6799",
      "name": "b119ec20-94fb-4593-816b-4d56620a6799",
      "status": "Failed",
      "startTime": "2026-10-22T02:22:58.803577295Z",
      "endTime": "2026-10-22T02:52:00.551152403Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, cluster-storage-operator, redhat-operators-catalog, capi-provider, cluster-network-operator, ingress-operator, olm-collect-profiles, openshift-controller-manager, azure-cloud-controller-manager, kube-storage-version-migrator, cluster-image-registry-operator, konnectivity-agent, machine-approver, kube-scheduler, cluster-version-operator, dns-operator, catalog-operator, cluster-autoscaler, control-plane-pki-operator, kube-controller-manager, openshift-route-controller-manager, cluster-policy-controller, olm-operator, packageserver, community-operators-catalog, certified-operators-catalog, openshift-apiserver; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b119ec20-94fb-4593-816b-4d56620a6799
            correlation ID: cb65955f-e919-441d-963c-27a6a9495a9b
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-5-0-bzlw46" with version 5.0.0-rc.6 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x202d14170440>: 
    failed to create HCP cluster cluster-zstream-5-0-bzlw46: failed waiting for cluster="cluster-zstream-5-0-bzlw46" in resourcegroup="rg-zstream-upgrade-5-0-l5ckf5pqgjzq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b119ec20-94fb-4593-816b-4d56620a6799
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b119ec20-94fb-4593-816b-4d56620a6799",
      "name": "b119ec20-94fb-4593-816b-4d56620a6799",
      "status": "Failed",
      "startTime": "2026-10-22T02:22:58.803577295Z",
      "endTime": "2026-10-22T02:52:00.551152403Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, cluster-storage-operator, redhat-operators-catalog, capi-provider, cluster-network-operator, ingress-operator, olm-collect-profiles, openshift-controller-manager, azure-cloud-controller-manager, kube-storage-version-migrator, cluster-image-registry-operator, konnectivity-agent, machine-approver, kube-scheduler, cluster-version-operator, dns-operator, catalog-operator, cluster-autoscaler, control-plane-pki-operator, kube-controller-manager, openshift-route-controller-manager, cluster-policy-controller, olm-operator, packageserver, community-operators-catalog, certified-operators-catalog, openshift-apiserver; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b119ec20-94fb-4593-816b-4d56620a6799
            correlation ID: cb65955f-e919-441d-963c-27a6a9495a9b
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-mszzh7" at install version 5.0
Unexpected error:
    <*fmt.wrapError | 0x11efeacc6c00>: 
    failed to create HCP cluster cp-ystream-upgrade-5-1-mszzh7: failed waiting for cluster="cp-ystream-upgrade-5-1-mszzh7" in resourcegroup="rg-cp-ystream-upgrade-5-1-w6jlwhdqxzx4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e60be32-8991-4fef-8fa2-b73c032aca4f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e60be32-8991-4fef-8fa2-b73c032aca4f",
      "name": "6e60be32-8991-4fef-8fa2-b73c032aca4f",
      "status": "Failed",
      "startTime": "2026-10-22T02:24:14.277639419Z",
      "endTime": "2026-10-22T02:53:20.714905396Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e60be32-8991-4fef-8fa2-b73c032aca4f
            correlation ID: 58eae255-dc55-44cf-aba8-2164cd64b6d5
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-mszzh7" at install version 5.0
Unexpected error:
    <*fmt.wrapError | 0x11efeacc6c00>: 
    failed to create HCP cluster cp-ystream-upgrade-5-1-mszzh7: failed waiting for cluster="cp-ystream-upgrade-5-1-mszzh7" in resourcegroup="rg-cp-ystream-upgrade-5-1-w6jlwhdqxzx4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e60be32-8991-4fef-8fa2-b73c032aca4f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e60be32-8991-4fef-8fa2-b73c032aca4f",
      "name": "6e60be32-8991-4fef-8fa2-b73c032aca4f",
      "status": "Failed",
      "startTime": "2026-10-22T02:24:14.277639419Z",
      "endTime": "2026-10-22T02:53:20.714905396Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e60be32-8991-4fef-8fa2-b73c032aca4f
            correlation ID: 58eae255-dc55-44cf-aba8-2164cd64b6d5
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xbec85184960>: 
    failed to create HCP cluster ea-listr985hw: failed waiting for cluster="ea-listr985hw" in resourcegroup="ea-list-psnhlv4lhlmk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d538c387-3fea-488a-a164-ab083059af27
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d538c387-3fea-488a-a164-ab083059af27",
      "name": "d538c387-3fea-488a-a164-ab083059af27",
      "status": "Failed",
      "startTime": "2026-10-22T02:25:04.906647307Z",
      "endTime": "2026-10-22T02:54:10.540852911Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d538c387-3fea-488a-a164-ab083059af27
            correlation ID: 924be768-33e9-47c5-9219-bd9273d15ea1
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xbec85184960>: 
    failed to create HCP cluster ea-listr985hw: failed waiting for cluster="ea-listr985hw" in resourcegroup="ea-list-psnhlv4lhlmk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d538c387-3fea-488a-a164-ab083059af27
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d538c387-3fea-488a-a164-ab083059af27",
      "name": "d538c387-3fea-488a-a164-ab083059af27",
      "status": "Failed",
      "startTime": "2026-10-22T02:25:04.906647307Z",
      "endTime": "2026-10-22T02:54:10.540852911Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d538c387-3fea-488a-a164-ab083059af27
            correlation ID: 924be768-33e9-47c5-9219-bd9273d15ea1
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should be able to retrieve cluster and nodepool status active versions · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:80]: failed to create HCP cluster "v20261001" via v20261001
Unexpected error:
    <*fmt.wrapError | 0x3a75f4e12f60>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001-fx25mzlnzj9t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f9269de-192b-41ce-bc41-692ff3dda38f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f9269de-192b-41ce-bc41-692ff3dda38f",
      "name": "0f9269de-192b-41ce-bc41-692ff3dda38f",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:16.144784385Z",
      "endTime": "2026-10-22T02:03:16.753815438Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f9269de-192b-41ce-bc41-692ff3dda38f
            correlation ID: 0c97842f-0279-4b59-9196-c2a2e8ab56cb
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:80]: failed to create HCP cluster "v20261001" via v20261001
Unexpected error:
    <*fmt.wrapError | 0x3a75f4e12f60>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001-fx25mzlnzj9t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f9269de-192b-41ce-bc41-692ff3dda38f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f9269de-192b-41ce-bc41-692ff3dda38f",
      "name": "0f9269de-192b-41ce-bc41-692ff3dda38f",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:16.144784385Z",
      "endTime": "2026-10-22T02:03:16.753815438Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f9269de-192b-41ce-bc41-692ff3dda38f
            correlation ID: 0c97842f-0279-4b59-9196-c2a2e8ab56cb
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should be able to cordon, drain, and uncordon a node in an HCP cluster · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:92]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x1c787f4ae5a0>: 
    failed to create HCP cluster cordon-hcp-cluster: failed waiting for cluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-s42wrnh68zb2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5da49df1-032c-48e7-8170-21123b884de0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5da49df1-032c-48e7-8170-21123b884de0",
      "name": "5da49df1-032c-48e7-8170-21123b884de0",
      "status": "Failed",
      "startTime": "2026-10-22T03:02:14.362284979Z",
      "endTime": "2026-10-22T03:31:15.143035388Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, konnectivity-agent, packageserver, control-plane-pki-operator, azure-cloud-controller-manager, machine-approver, cluster-version-operator, olm-operator, redhat-operators-catalog, redhat-marketplace-catalog, hosted-cluster-config-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, openshift-controller-manager, openshift-route-controller-manager, cluster-policy-controller, cluster-image-registry-operator, catalog-operator, community-operators-catalog, certified-operators-catalog, cluster-network-operator, dns-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5da49df1-032c-48e7-8170-21123b884de0
            correlation ID: 8ab168b3-b3fc-4668-8ddc-5c5d1b6b46a0
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:92]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x1c787f4ae5a0>: 
    failed to create HCP cluster cordon-hcp-cluster: failed waiting for cluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-s42wrnh68zb2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5da49df1-032c-48e7-8170-21123b884de0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5da49df1-032c-48e7-8170-21123b884de0",
      "name": "5da49df1-032c-48e7-8170-21123b884de0",
      "status": "Failed",
      "startTime": "2026-10-22T03:02:14.362284979Z",
      "endTime": "2026-10-22T03:31:15.143035388Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, konnectivity-agent, packageserver, control-plane-pki-operator, azure-cloud-controller-manager, machine-approver, cluster-version-operator, olm-operator, redhat-operators-catalog, redhat-marketplace-catalog, hosted-cluster-config-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, openshift-controller-manager, openshift-route-controller-manager, cluster-policy-controller, cluster-image-registry-operator, catalog-operator, community-operators-catalog, certified-operators-catalog, cluster-network-operator, dns-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5da49df1-032c-48e7-8170-21123b884de0
            correlation ID: 8ab168b3-b3fc-4668-8ddc-5c5d1b6b46a0
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0x383d8a0a0760>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-4hpbwr6m2mc9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fb9bf18a-3669-464c-b675-17049049d397
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fb9bf18a-3669-464c-b675-17049049d397",
      "name": "fb9bf18a-3669-464c-b675-17049049d397",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:04.492681455Z",
      "endTime": "2026-10-22T02:03:06.687321625Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fb9bf18a-3669-464c-b675-17049049d397
            correlation ID: fd8633cb-5d10-4988-961a-d49a1e752197
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0x383d8a0a0760>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-4hpbwr6m2mc9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fb9bf18a-3669-464c-b675-17049049d397
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fb9bf18a-3669-464c-b675-17049049d397",
      "name": "fb9bf18a-3669-464c-b675-17049049d397",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:04.492681455Z",
      "endTime": "2026-10-22T02:03:06.687321625Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fb9bf18a-3669-464c-b675-17049049d397
            correlation ID: fd8633cb-5d10-4988-961a-d49a1e752197
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver, router]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-4-23-d44bvbtv22rz/cluster-candidate-4-23-f2rpqw should provision
Unexpected error:
    <*fmt.wrapError | 0x126c5cc0e60>: 
    failed to create HCP cluster cluster-candidate-4-23-f2rpqw: failed waiting for cluster="cluster-candidate-4-23-f2rpqw" in resourcegroup="rg-candidate-4-23-d44bvbtv22rz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/efe747a0-4a74-4970-aae9-c5a7e67a1f5b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/efe747a0-4a74-4970-aae9-c5a7e67a1f5b",
      "name": "efe747a0-4a74-4970-aae9-c5a7e67a1f5b",
      "status": "Failed",
      "startTime": "2026-10-22T02:21:35.82450299Z",
      "endTime": "2026-10-22T02:50:40.503091649Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, catalog-operator, redhat-operators-catalog, machine-approver, openshift-controller-manager, konnectivity-agent, olm-operator, certified-operators-catalog, kube-controller-manager, cluster-version-operator, azure-cloud-controller-manager, kube-storage-version-migrator, community-operators-catalog, openshift-apiserver, cluster-policy-controller, cluster-autoscaler, cluster-network-operator, dns-operator, control-plane-pki-operator, kube-scheduler, hosted-cluster-config-operator, cluster-storage-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, packageserver, ingress-operator, cluster-image-registry-operator, olm-collect-profiles; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/efe747a0-4a74-4970-aae9-c5a7e67a1f5b
            correlation ID: 08db8465-69b5-4f99-aabf-f55441aa0ace
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-4-23-d44bvbtv22rz/cluster-candidate-4-23-f2rpqw should provision
Unexpected error:
    <*fmt.wrapError | 0x126c5cc0e60>: 
    failed to create HCP cluster cluster-candidate-4-23-f2rpqw: failed waiting for cluster="cluster-candidate-4-23-f2rpqw" in resourcegroup="rg-candidate-4-23-d44bvbtv22rz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/efe747a0-4a74-4970-aae9-c5a7e67a1f5b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/efe747a0-4a74-4970-aae9-c5a7e67a1f5b",
      "name": "efe747a0-4a74-4970-aae9-c5a7e67a1f5b",
      "status": "Failed",
      "startTime": "2026-10-22T02:21:35.82450299Z",
      "endTime": "2026-10-22T02:50:40.503091649Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, catalog-operator, redhat-operators-catalog, machine-approver, openshift-controller-manager, konnectivity-agent, olm-operator, certified-operators-catalog, kube-controller-manager, cluster-version-operator, azure-cloud-controller-manager, kube-storage-version-migrator, community-operators-catalog, openshift-apiserver, cluster-policy-controller, cluster-autoscaler, cluster-network-operator, dns-operator, control-plane-pki-operator, kube-scheduler, hosted-cluster-config-operator, cluster-storage-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, packageserver, ingress-operator, cluster-image-registry-operator, olm-collect-profiles; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/efe747a0-4a74-4970-aae9-c5a7e67a1f5b
            correlation ID: 08db8465-69b5-4f99-aabf-f55441aa0ace
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Nodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:127]: failed to create HCP cluster des-encrypt
Unexpected error:
    <*fmt.wrapError | 0x119df757e3e0>: 
    failed to create HCP cluster des-encrypt: failed waiting for cluster="des-encrypt" in resourcegroup="des-encrypt-hzm47sjhc426" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a6c258e-71e6-4920-8857-17ab7dc1e45f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a6c258e-71e6-4920-8857-17ab7dc1e45f",
      "name": "6a6c258e-71e6-4920-8857-17ab7dc1e45f",
      "status": "Failed",
      "startTime": "2026-10-22T03:09:48.207242699Z",
      "endTime": "2026-10-22T03:38:55.0934354Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, dns-operator, hosted-cluster-config-operator, azure-cloud-controller-manager, openshift-controller-manager, cluster-node-tuning-operator, catalog-operator, cluster-storage-operator, cluster-network-operator, cluster-image-registry-operator, certified-operators-catalog, csi-snapshot-controller-operator, konnectivity-agent, community-operators-catalog, kube-controller-manager, kube-scheduler, olm-operator, redhat-marketplace-catalog, openshift-route-controller-manager, cluster-policy-controller, packageserver, redhat-operators-catalog, control-plane-pki-operator, ingress-operator, machine-approver; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a6c258e-71e6-4920-8857-17ab7dc1e45f
            correlation ID: 80334f53-9a3b-4203-a7f4-9d0844089824
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:127]: failed to create HCP cluster des-encrypt
Unexpected error:
    <*fmt.wrapError | 0x119df757e3e0>: 
    failed to create HCP cluster des-encrypt: failed waiting for cluster="des-encrypt" in resourcegroup="des-encrypt-hzm47sjhc426" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a6c258e-71e6-4920-8857-17ab7dc1e45f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a6c258e-71e6-4920-8857-17ab7dc1e45f",
      "name": "6a6c258e-71e6-4920-8857-17ab7dc1e45f",
      "status": "Failed",
      "startTime": "2026-10-22T03:09:48.207242699Z",
      "endTime": "2026-10-22T03:38:55.0934354Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, dns-operator, hosted-cluster-config-operator, azure-cloud-controller-manager, openshift-controller-manager, cluster-node-tuning-operator, catalog-operator, cluster-storage-operator, cluster-network-operator, cluster-image-registry-operator, certified-operators-catalog, csi-snapshot-controller-operator, konnectivity-agent, community-operators-catalog, kube-controller-manager, kube-scheduler, olm-operator, redhat-marketplace-catalog, openshift-route-controller-manager, cluster-policy-controller, packageserver, redhat-operators-catalog, control-plane-pki-operator, ingress-operator, machine-approver; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a6c258e-71e6-4920-8857-17ab7dc1e45f
            correlation ID: 80334f53-9a3b-4203-a7f4-9d0844089824
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0x9e2f3a4840>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/579a5520-85b0-4d76-8f4f-8f11c4f8426f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/579a5520-85b0-4d76-8f4f-8f11c4f8426f",
      "name": "579a5520-85b0-4d76-8f4f-8f11c4f8426f",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:51.321199008Z",
      "endTime": "2026-10-22T02:03:56.79400182Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-apiserver, cluster-policy-controller, azure-cloud-controller-manager, catalog-operator, redhat-marketplace-catalog, certified-operators-catalog, control-plane-pki-operator, hosted-cluster-config-operator, cluster-storage-operator, cluster-network-operator, cluster-node-tuning-operator, dns-operator, ingress-operator, cluster-image-registry-operator, kube-scheduler, openshift-controller-manager, konnectivity-agent, olm-operator, packageserver, olm-collect-profiles, redhat-operators-catalog, cluster-autoscaler, kube-controller-manager, cluster-version-operator, openshift-route-controller-manager, machine-approver, csi-snapshot-controller-operator, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/579a5520-85b0-4d76-8f4f-8f11c4f8426f
            correlation ID: 02b44ad8-cedb-41d0-85b1-5d72c2719877
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0x9e2f3a4840>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/579a5520-85b0-4d76-8f4f-8f11c4f8426f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/579a5520-85b0-4d76-8f4f-8f11c4f8426f",
      "name": "579a5520-85b0-4d76-8f4f-8f11c4f8426f",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:51.321199008Z",
      "endTime": "2026-10-22T02:03:56.79400182Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-apiserver, cluster-policy-controller, azure-cloud-controller-manager, catalog-operator, redhat-marketplace-catalog, certified-operators-catalog, control-plane-pki-operator, hosted-cluster-config-operator, cluster-storage-operator, cluster-network-operator, cluster-node-tuning-operator, dns-operator, ingress-operator, cluster-image-registry-operator, kube-scheduler, openshift-controller-manager, konnectivity-agent, olm-operator, packageserver, olm-collect-profiles, redhat-operators-catalog, cluster-autoscaler, kube-controller-manager, cluster-version-operator, openshift-route-controller-manager, machine-approver, csi-snapshot-controller-operator, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/579a5520-85b0-4d76-8f4f-8f11c4f8426f
            correlation ID: 02b44ad8-cedb-41d0-85b1-5d72c2719877
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xaf2b1adc460>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-clfhrsrccp5c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2f16d2a2-0de5-4840-9b52-f254619dcdd9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2f16d2a2-0de5-4840-9b52-f254619dcdd9",
      "name": "2f16d2a2-0de5-4840-9b52-f254619dcdd9",
      "status": "Failed",
      "startTime": "2026-10-22T02:27:00.294362099Z",
      "endTime": "2026-10-22T02:56:00.372773919Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2f16d2a2-0de5-4840-9b52-f254619dcdd9
            correlation ID: 8e6afc06-e48b-4d7e-a53d-fbab147992fd
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xaf2b1adc460>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-clfhrsrccp5c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2f16d2a2-0de5-4840-9b52-f254619dcdd9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2f16d2a2-0de5-4840-9b52-f254619dcdd9",
      "name": "2f16d2a2-0de5-4840-9b52-f254619dcdd9",
      "status": "Failed",
      "startTime": "2026-10-22T02:27:00.294362099Z",
      "endTime": "2026-10-22T02:56:00.372773919Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2f16d2a2-0de5-4840-9b52-f254619dcdd9
            correlation ID: 8e6afc06-e48b-4d7e-a53d-fbab147992fd
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [openshift-route-controller-manager]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x3ede8eb28c00>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-qzb5xz65mghw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/43a33130-1505-4488-8d2c-b0f13de47cff
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/43a33130-1505-4488-8d2c-b0f13de47cff",
      "name": "43a33130-1505-4488-8d2c-b0f13de47cff",
      "status": "Failed",
      "startTime": "2026-10-22T03:04:08.329961738Z",
      "endTime": "2026-10-22T03:33:15.140670523Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: openshift-route-controller-manager deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/43a33130-1505-4488-8d2c-b0f13de47cff
            correlation ID: 2c680bb1-7f2a-45c1-a484-bbb38ed78677
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x3ede8eb28c00>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-qzb5xz65mghw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/43a33130-1505-4488-8d2c-b0f13de47cff
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/43a33130-1505-4488-8d2c-b0f13de47cff",
      "name": "43a33130-1505-4488-8d2c-b0f13de47cff",
      "status": "Failed",
      "startTime": "2026-10-22T03:04:08.329961738Z",
      "endTime": "2026-10-22T03:33:15.140670523Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: openshift-route-controller-manager deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/43a33130-1505-4488-8d2c-b0f13de47cff
            correlation ID: 2c680bb1-7f2a-45c1-a484-bbb38ed78677
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:98]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0x33ca21720880>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-6zgdp7hhpk7k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c707ea6e-b74b-443a-9b32-fe7f80a738f2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c707ea6e-b74b-443a-9b32-fe7f80a738f2",
      "name": "c707ea6e-b74b-443a-9b32-fe7f80a738f2",
      "status": "Failed",
      "startTime": "2026-10-22T02:25:00.37199898Z",
      "endTime": "2026-10-22T02:54:00.386898896Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c707ea6e-b74b-443a-9b32-fe7f80a738f2
            correlation ID: cd0ac0e6-4a75-4391-a2be-e65722141c43
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:98]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0x33ca21720880>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-6zgdp7hhpk7k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c707ea6e-b74b-443a-9b32-fe7f80a738f2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c707ea6e-b74b-443a-9b32-fe7f80a738f2",
      "name": "c707ea6e-b74b-443a-9b32-fe7f80a738f2",
      "status": "Failed",
      "startTime": "2026-10-22T02:25:00.37199898Z",
      "endTime": "2026-10-22T02:54:00.386898896Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c707ea6e-b74b-443a-9b32-fe7f80a738f2
            correlation ID: cd0ac0e6-4a75-4391-a2be-e65722141c43
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [capi-provider, kube-apiserver]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-dsklst" with version 4.22.18 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x19639643c920>: 
    failed to create HCP cluster cluster-zstream-4-22-dsklst: failed waiting for cluster="cluster-zstream-4-22-dsklst" in resourcegroup="rg-zstream-upgrade-4-22-5t2tjvkj99ps" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f8c2f6-a8c6-4ec9-bef8-08b6e4a75a38
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f8c2f6-a8c6-4ec9-bef8-08b6e4a75a38",
      "name": "13f8c2f6-a8c6-4ec9-bef8-08b6e4a75a38",
      "status": "Failed",
      "startTime": "2026-10-22T02:22:46.501521761Z",
      "endTime": "2026-10-22T02:51:50.360464178Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, ingress-operator, cluster-image-registry-operator, community-operators-catalog, machine-approver, azure-cloud-controller-manager, dns-operator, hosted-cluster-config-operator, csi-snapshot-controller-operator, packageserver, openshift-controller-manager, konnectivity-agent, certified-operators-catalog, kube-storage-version-migrator, catalog-operator, capi-provider, openshift-apiserver, openshift-route-controller-manager, cluster-storage-operator, cluster-network-operator, cluster-autoscaler, cluster-policy-controller, kube-scheduler, olm-collect-profiles, redhat-operators-catalog, control-plane-pki-operator, kube-controller-manager, cluster-node-tuning-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f8c2f6-a8c6-4ec9-bef8-08b6e4a75a38
            correlation ID: 4f330ac9-7640-45b9-b580-8c05eae65daf
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-dsklst" with version 4.22.18 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x19639643c920>: 
    failed to create HCP cluster cluster-zstream-4-22-dsklst: failed waiting for cluster="cluster-zstream-4-22-dsklst" in resourcegroup="rg-zstream-upgrade-4-22-5t2tjvkj99ps" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f8c2f6-a8c6-4ec9-bef8-08b6e4a75a38
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f8c2f6-a8c6-4ec9-bef8-08b6e4a75a38",
      "name": "13f8c2f6-a8c6-4ec9-bef8-08b6e4a75a38",
      "status": "Failed",
      "startTime": "2026-10-22T02:22:46.501521761Z",
      "endTime": "2026-10-22T02:51:50.360464178Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, ingress-operator, cluster-image-registry-operator, community-operators-catalog, machine-approver, azure-cloud-controller-manager, dns-operator, hosted-cluster-config-operator, csi-snapshot-controller-operator, packageserver, openshift-controller-manager, konnectivity-agent, certified-operators-catalog, kube-storage-version-migrator, catalog-operator, capi-provider, openshift-apiserver, openshift-route-controller-manager, cluster-storage-operator, cluster-network-operator, cluster-autoscaler, cluster-policy-controller, kube-scheduler, olm-collect-profiles, redhat-operators-catalog, control-plane-pki-operator, kube-controller-manager, cluster-node-tuning-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f8c2f6-a8c6-4ec9-bef8-08b6e4a75a38
            correlation ID: 4f330ac9-7640-45b9-b580-8c05eae65daf
            client request ID: unavailable
occurred
Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed
01:09:23 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0x19104d7cc9c0>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-l8cntnddrw7h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/990ea455-66b6-4073-a5f2-4aeddb743877
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/990ea455-66b6-4073-a5f2-4aeddb743877",
      "name": "990ea455-66b6-4073-a5f2-4aeddb743877",
      "status": "Failed",
      "startTime": "2026-10-22T01:36:21.537163403Z",
      "endTime": "2026-10-22T02:05:26.659061975Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/990ea455-66b6-4073-a5f2-4aeddb743877
            correlation ID: 3c80a876-0ab2-4b84-be94-2ce70586894c
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0x19104d7cc9c0>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-l8cntnddrw7h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/990ea455-66b6-4073-a5f2-4aeddb743877
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/990ea455-66b6-4073-a5f2-4aeddb743877",
      "name": "990ea455-66b6-4073-a5f2-4aeddb743877",
      "status": "Failed",
      "startTime": "2026-10-22T01:36:21.537163403Z",
      "endTime": "2026-10-22T02:05:26.659061975Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/990ea455-66b6-4073-a5f2-4aeddb743877
            correlation ID: 3c80a876-0ab2-4b84-be94-2ce70586894c
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, konnectivity-agent, kube-controller-manager, kube-scheduler, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-pp2v6j" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0x9f443c1ebc0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-pp2v6j: failed waiting for cluster="cp-ystream-upgrade-4-22-pp2v6j" in resourcegroup="rg-cp-ystream-upgrade-4-22-4brmx258wcvt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c465a539-e0e2-4a15-96b0-b991fee60416
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c465a539-e0e2-4a15-96b0-b991fee60416",
      "name": "c465a539-e0e2-4a15-96b0-b991fee60416",
      "status": "Failed",
      "startTime": "2026-10-22T02:23:40.34684753Z",
      "endTime": "2026-10-22T02:52:40.362346793Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, cluster-storage-operator, cluster-policy-controller, packageserver, hosted-cluster-config-operator, cluster-image-registry-operator, kube-controller-manager, cluster-version-operator, community-operators-catalog, openshift-controller-manager, cluster-node-tuning-operator, redhat-operators-catalog, redhat-marketplace-catalog, certified-operators-catalog, openshift-route-controller-manager, kube-scheduler, cluster-network-operator, dns-operator, konnectivity-agent, olm-operator, ingress-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c465a539-e0e2-4a15-96b0-b991fee60416
            correlation ID: 81f12348-0a31-441f-af5f-8f18eef28715
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-pp2v6j" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0x9f443c1ebc0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-pp2v6j: failed waiting for cluster="cp-ystream-upgrade-4-22-pp2v6j" in resourcegroup="rg-cp-ystream-upgrade-4-22-4brmx258wcvt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c465a539-e0e2-4a15-96b0-b991fee60416
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c465a539-e0e2-4a15-96b0-b991fee60416",
      "name": "c465a539-e0e2-4a15-96b0-b991fee60416",
      "status": "Failed",
      "startTime": "2026-10-22T02:23:40.34684753Z",
      "endTime": "2026-10-22T02:52:40.362346793Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, cluster-storage-operator, cluster-policy-controller, packageserver, hosted-cluster-config-operator, cluster-image-registry-operator, kube-controller-manager, cluster-version-operator, community-operators-catalog, openshift-controller-manager, cluster-node-tuning-operator, redhat-operators-catalog, redhat-marketplace-catalog, certified-operators-catalog, openshift-route-controller-manager, kube-scheduler, cluster-network-operator, dns-operator, konnectivity-agent, olm-operator, ingress-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c465a539-e0e2-4a15-96b0-b991fee60416
            correlation ID: 81f12348-0a31-441f-af5f-8f18eef28715
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0x363c02e8080>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-jbrqpz7ctxtq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/21e6bfc0-fe4d-4dac-b056-c840f710f982
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/21e6bfc0-fe4d-4dac-b056-c840f710f982",
      "name": "21e6bfc0-fe4d-4dac-b056-c840f710f982",
      "status": "Failed",
      "startTime": "2026-10-22T03:11:29.420591363Z",
      "endTime": "2026-10-22T03:40:35.058626079Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/21e6bfc0-fe4d-4dac-b056-c840f710f982
            correlation ID: c21da3fc-8f7d-45b1-bbc4-5d3a6ee72767
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0x363c02e8080>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-jbrqpz7ctxtq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/21e6bfc0-fe4d-4dac-b056-c840f710f982
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/21e6bfc0-fe4d-4dac-b056-c840f710f982",
      "name": "21e6bfc0-fe4d-4dac-b056-c840f710f982",
      "status": "Failed",
      "startTime": "2026-10-22T03:11:29.420591363Z",
      "endTime": "2026-10-22T03:40:35.058626079Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/21e6bfc0-fe4d-4dac-b056-c840f710f982
            correlation ID: c21da3fc-8f7d-45b1-bbc4-5d3a6ee72767
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-5-1-2s5zlgwb6bct/cluster-candidate-5-1-84zdqd should provision
Unexpected error:
    <*fmt.wrapError | 0x16cd8e8f9880>: 
    failed to create HCP cluster cluster-candidate-5-1-84zdqd: failed waiting for cluster="cluster-candidate-5-1-84zdqd" in resourcegroup="rg-candidate-5-1-2s5zlgwb6bct" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9d897978-69ec-403d-ad9b-fd32c33747d4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9d897978-69ec-403d-ad9b-fd32c33747d4",
      "name": "9d897978-69ec-403d-ad9b-fd32c33747d4",
      "status": "Failed",
      "startTime": "2026-10-22T02:21:55.892991844Z",
      "endTime": "2026-10-22T02:51:00.378883542Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: machine-approver, csi-snapshot-controller-operator, cluster-image-registry-operator, catalog-operator, kube-scheduler, dns-operator, konnectivity-agent, kube-controller-manager, cluster-storage-operator, cluster-autoscaler, cluster-policy-controller, hosted-cluster-config-operator, olm-operator, olm-collect-profiles, community-operators-catalog, cluster-version-operator, kube-storage-version-migrator, packageserver, control-plane-pki-operator, openshift-controller-manager, azure-cloud-controller-manager, openshift-apiserver, openshift-route-controller-manager, redhat-operators-catalog, certified-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9d897978-69ec-403d-ad9b-fd32c33747d4
            correlation ID: bdf3de8a-4b44-40d3-b952-4e50b1bc65e3
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-5-1-2s5zlgwb6bct/cluster-candidate-5-1-84zdqd should provision
Unexpected error:
    <*fmt.wrapError | 0x16cd8e8f9880>: 
    failed to create HCP cluster cluster-candidate-5-1-84zdqd: failed waiting for cluster="cluster-candidate-5-1-84zdqd" in resourcegroup="rg-candidate-5-1-2s5zlgwb6bct" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9d897978-69ec-403d-ad9b-fd32c33747d4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9d897978-69ec-403d-ad9b-fd32c33747d4",
      "name": "9d897978-69ec-403d-ad9b-fd32c33747d4",
      "status": "Failed",
      "startTime": "2026-10-22T02:21:55.892991844Z",
      "endTime": "2026-10-22T02:51:00.378883542Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: machine-approver, csi-snapshot-controller-operator, cluster-image-registry-operator, catalog-operator, kube-scheduler, dns-operator, konnectivity-agent, kube-controller-manager, cluster-storage-operator, cluster-autoscaler, cluster-policy-controller, hosted-cluster-config-operator, olm-operator, olm-collect-profiles, community-operators-catalog, cluster-version-operator, kube-storage-version-migrator, packageserver, control-plane-pki-operator, openshift-controller-manager, azure-cloud-controller-manager, openshift-apiserver, openshift-route-controller-manager, redhat-operators-catalog, certified-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9d897978-69ec-403d-ad9b-fd32c33747d4
            correlation ID: bdf3de8a-4b44-40d3-b952-4e50b1bc65e3
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-fvrddf at version 4.22.19
Unexpected error:
    <*fmt.wrapError | 0x3b64a153a380>: 
    failed to create HCP cluster np-skip-minor-fvrddf: failed waiting for cluster="np-skip-minor-fvrddf" in resourcegroup="rg-np-skip-minor-fvrddf-fpw64csf8rm6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fcd2c383-b881-499c-8bed-cb24dce8c319
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fcd2c383-b881-499c-8bed-cb24dce8c319",
      "name": "fcd2c383-b881-499c-8bed-cb24dce8c319",
      "status": "Failed",
      "startTime": "2026-10-22T03:10:54.281108118Z",
      "endTime": "2026-10-22T03:39:55.15624633Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fcd2c383-b881-499c-8bed-cb24dce8c319
            correlation ID: e6ae3233-540c-4141-81e8-0b055bb45703
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-fvrddf at version 4.22.19
Unexpected error:
    <*fmt.wrapError | 0x3b64a153a380>: 
    failed to create HCP cluster np-skip-minor-fvrddf: failed waiting for cluster="np-skip-minor-fvrddf" in resourcegroup="rg-np-skip-minor-fvrddf-fpw64csf8rm6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fcd2c383-b881-499c-8bed-cb24dce8c319
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fcd2c383-b881-499c-8bed-cb24dce8c319",
      "name": "fcd2c383-b881-499c-8bed-cb24dce8c319",
      "status": "Failed",
      "startTime": "2026-10-22T03:10:54.281108118Z",
      "endTime": "2026-10-22T03:39:55.15624633Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fcd2c383-b881-499c-8bed-cb24dce8c319
            correlation ID: e6ae3233-540c-4141-81e8-0b055bb45703
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:163]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0x2d4d33b70620>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-b5r6x68bzp5t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a651d305-0bc4-4e50-8659-237a66cb9f85
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a651d305-0bc4-4e50-8659-237a66cb9f85",
      "name": "a651d305-0bc4-4e50-8659-237a66cb9f85",
      "status": "Failed",
      "startTime": "2026-10-22T02:20:28.664519088Z",
      "endTime": "2026-10-22T02:49:30.367118005Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, control-plane-pki-operator, catalog-operator, olm-operator, openshift-controller-manager, hosted-cluster-config-operator, azure-cloud-controller-manager, cluster-network-operator, cluster-policy-controller, cluster-image-registry-operator, redhat-operators-catalog, redhat-marketplace-catalog, cluster-node-tuning-operator, machine-approver, dns-operator, ingress-operator, community-operators-catalog, kube-controller-manager, cluster-storage-operator, csi-snapshot-controller-operator, konnectivity-agent, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a651d305-0bc4-4e50-8659-237a66cb9f85
            correlation ID: 1caf4b05-a329-41fc-bcd5-7a64b3705d89
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:163]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0x2d4d33b70620>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-b5r6x68bzp5t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a651d305-0bc4-4e50-8659-237a66cb9f85
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a651d305-0bc4-4e50-8659-237a66cb9f85",
      "name": "a651d305-0bc4-4e50-8659-237a66cb9f85",
      "status": "Failed",
      "startTime": "2026-10-22T02:20:28.664519088Z",
      "endTime": "2026-10-22T02:49:30.367118005Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, control-plane-pki-operator, catalog-operator, olm-operator, openshift-controller-manager, hosted-cluster-config-operator, azure-cloud-controller-manager, cluster-network-operator, cluster-policy-controller, cluster-image-registry-operator, redhat-operators-catalog, redhat-marketplace-catalog, cluster-node-tuning-operator, machine-approver, dns-operator, ingress-operator, community-operators-catalog, kube-controller-manager, cluster-storage-operator, csi-snapshot-controller-operator, konnectivity-agent, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a651d305-0bc4-4e50-8659-237a66cb9f85
            correlation ID: 1caf4b05-a329-41fc-bcd5-7a64b3705d89
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server-proxy, router]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-4-21-76q9tvzg8rn9/cluster-candidate-4-21-5lg29k should provision
Unexpected error:
    <*fmt.wrapError | 0x143589e84400>: 
    failed to create HCP cluster cluster-candidate-4-21-5lg29k: failed waiting for cluster="cluster-candidate-4-21-5lg29k" in resourcegroup="rg-candidate-4-21-76q9tvzg8rn9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edb6a2fa-979a-4af5-a995-cd3897780e01
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edb6a2fa-979a-4af5-a995-cd3897780e01",
      "name": "edb6a2fa-979a-4af5-a995-cd3897780e01",
      "status": "Failed",
      "startTime": "2026-10-22T02:21:35.877670552Z",
      "endTime": "2026-10-22T02:50:40.514192951Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edb6a2fa-979a-4af5-a995-cd3897780e01
            correlation ID: 85af8528-9942-40bd-9e4c-05e55063f9c4
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-4-21-76q9tvzg8rn9/cluster-candidate-4-21-5lg29k should provision
Unexpected error:
    <*fmt.wrapError | 0x143589e84400>: 
    failed to create HCP cluster cluster-candidate-4-21-5lg29k: failed waiting for cluster="cluster-candidate-4-21-5lg29k" in resourcegroup="rg-candidate-4-21-76q9tvzg8rn9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edb6a2fa-979a-4af5-a995-cd3897780e01
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edb6a2fa-979a-4af5-a995-cd3897780e01",
      "name": "edb6a2fa-979a-4af5-a995-cd3897780e01",
      "status": "Failed",
      "startTime": "2026-10-22T02:21:35.877670552Z",
      "endTime": "2026-10-22T02:50:40.514192951Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edb6a2fa-979a-4af5-a995-cd3897780e01
            correlation ID: 85af8528-9942-40bd-9e4c-05e55063f9c4
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test ARO-HCP should serve the default ingress certificate for 5.1 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:103]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x1fa7bf968ac0>: 
    failed to create HCP cluster default-ingress-cert: failed waiting for cluster="default-ingress-cert" in resourcegroup="ingress-cert-le-dw92nhksl48m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/956158f8-f657-469c-a6c1-31c01e3a3b19
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/956158f8-f657-469c-a6c1-31c01e3a3b19",
      "name": "956158f8-f657-469c-a6c1-31c01e3a3b19",
      "status": "Failed",
      "startTime": "2026-10-22T02:25:05.287575354Z",
      "endTime": "2026-10-22T02:54:10.367808601Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/956158f8-f657-469c-a6c1-31c01e3a3b19
            correlation ID: 70395257-9eb1-4aa1-8773-0b99bd5a9b69
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:103]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x1fa7bf968ac0>: 
    failed to create HCP cluster default-ingress-cert: failed waiting for cluster="default-ingress-cert" in resourcegroup="ingress-cert-le-dw92nhksl48m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/956158f8-f657-469c-a6c1-31c01e3a3b19
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/956158f8-f657-469c-a6c1-31c01e3a3b19",
      "name": "956158f8-f657-469c-a6c1-31c01e3a3b19",
      "status": "Failed",
      "startTime": "2026-10-22T02:25:05.287575354Z",
      "endTime": "2026-10-22T02:54:10.367808601Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/956158f8-f657-469c-a6c1-31c01e3a3b19
            correlation ID: 70395257-9eb1-4aa1-8773-0b99bd5a9b69
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, cluster-api, ignition-server]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-5-0-gmftj87c69s6/cluster-candidate-5-0-25snrj should provision
Unexpected error:
    <*fmt.wrapError | 0x1ed3b41bc940>: 
    failed to create HCP cluster cluster-candidate-5-0-25snrj: failed waiting for cluster="cluster-candidate-5-0-25snrj" in resourcegroup="rg-candidate-5-0-gmftj87c69s6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/27425e0a-5b12-47d6-9369-3a3a90aa3e86
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/27425e0a-5b12-47d6-9369-3a3a90aa3e86",
      "name": "27425e0a-5b12-47d6-9369-3a3a90aa3e86",
      "status": "Failed",
      "startTime": "2026-10-22T02:21:48.683878735Z",
      "endTime": "2026-10-22T02:50:50.395326627Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, cluster-api deployment has 2 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/27425e0a-5b12-47d6-9369-3a3a90aa3e86
            correlation ID: 52162704-0f0b-4a93-a71e-0f363b9222b6
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-5-0-gmftj87c69s6/cluster-candidate-5-0-25snrj should provision
Unexpected error:
    <*fmt.wrapError | 0x1ed3b41bc940>: 
    failed to create HCP cluster cluster-candidate-5-0-25snrj: failed waiting for cluster="cluster-candidate-5-0-25snrj" in resourcegroup="rg-candidate-5-0-gmftj87c69s6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/27425e0a-5b12-47d6-9369-3a3a90aa3e86
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/27425e0a-5b12-47d6-9369-3a3a90aa3e86",
      "name": "27425e0a-5b12-47d6-9369-3a3a90aa3e86",
      "status": "Failed",
      "startTime": "2026-10-22T02:21:48.683878735Z",
      "endTime": "2026-10-22T02:50:50.395326627Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, cluster-api deployment has 2 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/27425e0a-5b12-47d6-9369-3a3a90aa3e86
            correlation ID: 52162704-0f0b-4a93-a71e-0f363b9222b6
            client request ID: unavailable
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
01:09:23 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0x146ba53e5a0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-vgggt2t7qcsd" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vgggt2t7qcsd, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-vgggt2t7qcsd\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vgggt2t7qcsd, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x146ba6ff790>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vgggt2t7qcsd",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0x146ba53e5a0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-vgggt2t7qcsd" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vgggt2t7qcsd, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-vgggt2t7qcsd\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vgggt2t7qcsd, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x146ba6ff790>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vgggt2t7qcsd",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server-proxy]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x1f5c59786c0>: 
    failed to create HCP cluster np-dg-minor-8ks925: failed waiting for cluster="np-dg-minor-8ks925" in resourcegroup="rg-np-dg-minor-8ks925-jgl47czbdtkx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c0ead9e-e84e-4227-ab86-537912304e63
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c0ead9e-e84e-4227-ab86-537912304e63",
      "name": "3c0ead9e-e84e-4227-ab86-537912304e63",
      "status": "Failed",
      "startTime": "2026-10-22T03:10:59.807393048Z",
      "endTime": "2026-10-22T03:40:05.155981258Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c0ead9e-e84e-4227-ab86-537912304e63
            correlation ID: f292fc5b-a9dd-4382-9a58-48e706cd1073
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x1f5c59786c0>: 
    failed to create HCP cluster np-dg-minor-8ks925: failed waiting for cluster="np-dg-minor-8ks925" in resourcegroup="rg-np-dg-minor-8ks925-jgl47czbdtkx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c0ead9e-e84e-4227-ab86-537912304e63
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c0ead9e-e84e-4227-ab86-537912304e63",
      "name": "3c0ead9e-e84e-4227-ab86-537912304e63",
      "status": "Failed",
      "startTime": "2026-10-22T03:10:59.807393048Z",
      "endTime": "2026-10-22T03:40:05.155981258Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c0ead9e-e84e-4227-ab86-537912304e63
            correlation ID: f292fc5b-a9dd-4382-9a58-48e706cd1073
            client request ID: unavailable
occurred
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x16e186f694b8>: 
    resource group "ephemeral-osdisk-k7jcdd2kqwvh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-k7jcdd2kqwvh" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36a51701-9d7a-4ad4-860f-d0b45481d5d2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36a51701-9d7a-4ad4-860f-d0b45481d5d2",
      "name": "36a51701-9d7a-4ad4-860f-d0b45481d5d2",
      "status": "Failed",
      "startTime": "2026-10-22T03:23:14.821436082Z",
      "endTime": "2026-10-22T03:47:22.914912373Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tjj0cikkek0bc3m05t20hehmct394q5 still exists (deletion dispatched at 2026-10-22T03:23:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36a51701-9d7a-4ad4-860f-d0b45481d5d2
            correlation ID: b94afee3-9d6b-4695-a5b0-3f0b5bdf76c0
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x16e186f694b8>: 
    resource group "ephemeral-osdisk-k7jcdd2kqwvh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-k7jcdd2kqwvh" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36a51701-9d7a-4ad4-860f-d0b45481d5d2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36a51701-9d7a-4ad4-860f-d0b45481d5d2",
      "name": "36a51701-9d7a-4ad4-860f-d0b45481d5d2",
      "status": "Failed",
      "startTime": "2026-10-22T03:23:14.821436082Z",
      "endTime": "2026-10-22T03:47:22.914912373Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tjj0cikkek0bc3m05t20hehmct394q5 still exists (deletion dispatched at 2026-10-22T03:23:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36a51701-9d7a-4ad4-860f-d0b45481d5d2
            correlation ID: b94afee3-9d6b-4695-a5b0-3f0b5bdf76c0
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0x133ad1a668c0>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-6f4gg27gdpsf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/100e0456-7bbd-4421-b2b7-99394dc2ddc5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/100e0456-7bbd-4421-b2b7-99394dc2ddc5",
      "name": "100e0456-7bbd-4421-b2b7-99394dc2ddc5",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:06.328419317Z",
      "endTime": "2026-10-22T02:03:06.759634076Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, openshift-controller-manager, cluster-policy-controller, dns-operator, community-operators-catalog, catalog-operator, csi-snapshot-controller-operator, azure-cloud-controller-manager, openshift-route-controller-manager, hosted-cluster-config-operator, olm-operator, certified-operators-catalog, machine-approver, control-plane-pki-operator, kube-controller-manager, cluster-image-registry-operator, redhat-marketplace-catalog, cluster-storage-operator, cluster-network-operator, cluster-node-tuning-operator, ingress-operator, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/100e0456-7bbd-4421-b2b7-99394dc2ddc5
            correlation ID: bee0e436-ae08-4d85-a183-4029b932065b
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0x133ad1a668c0>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-6f4gg27gdpsf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/100e0456-7bbd-4421-b2b7-99394dc2ddc5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/100e0456-7bbd-4421-b2b7-99394dc2ddc5",
      "name": "100e0456-7bbd-4421-b2b7-99394dc2ddc5",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:06.328419317Z",
      "endTime": "2026-10-22T02:03:06.759634076Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, openshift-controller-manager, cluster-policy-controller, dns-operator, community-operators-catalog, catalog-operator, csi-snapshot-controller-operator, azure-cloud-controller-manager, openshift-route-controller-manager, hosted-cluster-config-operator, olm-operator, certified-operators-catalog, machine-approver, control-plane-pki-operator, kube-controller-manager, cluster-image-registry-operator, redhat-marketplace-catalog, cluster-storage-operator, cluster-network-operator, cluster-node-tuning-operator, ingress-operator, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/100e0456-7bbd-4421-b2b7-99394dc2ddc5
            correlation ID: bee0e436-ae08-4d85-a183-4029b932065b
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0x2951a1afed40>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-hfpdp8lmxh49" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0119dd29-201c-4c22-9ebd-7d3a5b8701fc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0119dd29-201c-4c22-9ebd-7d3a5b8701fc",
      "name": "0119dd29-201c-4c22-9ebd-7d3a5b8701fc",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:06.130621039Z",
      "endTime": "2026-10-22T02:03:06.668308317Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-collect-profiles, openshift-apiserver, openshift-route-controller-manager, dns-operator, csi-snapshot-controller-operator, kube-scheduler, cluster-policy-controller, packageserver, certified-operators-catalog, hosted-cluster-config-operator, cluster-version-operator, openshift-controller-manager, olm-operator, community-operators-catalog, machine-approver, control-plane-pki-operator, cluster-network-operator, konnectivity-agent, kube-storage-version-migrator, catalog-operator, cluster-storage-operator, redhat-operators-catalog, cluster-autoscaler, kube-controller-manager, azure-cloud-controller-manager, cluster-node-tuning-operator, ingress-operator, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0119dd29-201c-4c22-9ebd-7d3a5b8701fc
            correlation ID: 909da2c2-f472-4c31-8509-d71d62f94fb1
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0x2951a1afed40>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-hfpdp8lmxh49" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0119dd29-201c-4c22-9ebd-7d3a5b8701fc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0119dd29-201c-4c22-9ebd-7d3a5b8701fc",
      "name": "0119dd29-201c-4c22-9ebd-7d3a5b8701fc",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:06.130621039Z",
      "endTime": "2026-10-22T02:03:06.668308317Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-collect-profiles, openshift-apiserver, openshift-route-controller-manager, dns-operator, csi-snapshot-controller-operator, kube-scheduler, cluster-policy-controller, packageserver, certified-operators-catalog, hosted-cluster-config-operator, cluster-version-operator, openshift-controller-manager, olm-operator, community-operators-catalog, machine-approver, control-plane-pki-operator, cluster-network-operator, konnectivity-agent, kube-storage-version-migrator, catalog-operator, cluster-storage-operator, redhat-operators-catalog, cluster-autoscaler, kube-controller-manager, azure-cloud-controller-manager, cluster-node-tuning-operator, ingress-operator, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0119dd29-201c-4c22-9ebd-7d3a5b8701fc
            correlation ID: 909da2c2-f472-4c31-8509-d71d62f94fb1
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 5.1 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:147]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x34aa3403e6c0>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-cqgmsxlrrppn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ee102d0-1601-47d2-8777-14ff5b026dea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ee102d0-1601-47d2-8777-14ff5b026dea",
      "name": "5ee102d0-1601-47d2-8777-14ff5b026dea",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:04.925762167Z",
      "endTime": "2026-10-22T02:03:06.678876471Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, cluster-storage-operator, cluster-image-registry-operator, certified-operators-catalog, kube-storage-version-migrator, machine-approver, kube-controller-manager, kube-scheduler, openshift-controller-manager, cluster-network-operator, redhat-operators-catalog, hosted-cluster-config-operator, dns-operator, csi-snapshot-controller-operator, openshift-route-controller-manager, azure-cloud-controller-manager, cluster-node-tuning-operator, olm-operator, packageserver, cluster-version-operator, ingress-operator, konnectivity-agent, community-operators-catalog, catalog-operator, control-plane-pki-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ee102d0-1601-47d2-8777-14ff5b026dea
            correlation ID: 3d6a20c9-050a-45db-a515-be5eb69c9b7e
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:147]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x34aa3403e6c0>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-cqgmsxlrrppn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ee102d0-1601-47d2-8777-14ff5b026dea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ee102d0-1601-47d2-8777-14ff5b026dea",
      "name": "5ee102d0-1601-47d2-8777-14ff5b026dea",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:04.925762167Z",
      "endTime": "2026-10-22T02:03:06.678876471Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, cluster-storage-operator, cluster-image-registry-operator, certified-operators-catalog, kube-storage-version-migrator, machine-approver, kube-controller-manager, kube-scheduler, openshift-controller-manager, cluster-network-operator, redhat-operators-catalog, hosted-cluster-config-operator, dns-operator, csi-snapshot-controller-operator, openshift-route-controller-manager, azure-cloud-controller-manager, cluster-node-tuning-operator, olm-operator, packageserver, cluster-version-operator, ingress-operator, konnectivity-agent, community-operators-catalog, catalog-operator, control-plane-pki-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ee102d0-1601-47d2-8777-14ff5b026dea
            correlation ID: 3d6a20c9-050a-45db-a515-be5eb69c9b7e
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0x39761e268740>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-dm68szdn2k7s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3da97f2b-8dc4-47f0-81ba-91f4190a2d02
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3da97f2b-8dc4-47f0-81ba-91f4190a2d02",
      "name": "3da97f2b-8dc4-47f0-81ba-91f4190a2d02",
      "status": "Failed",
      "startTime": "2026-10-22T02:26:51.272492265Z",
      "endTime": "2026-10-22T02:56:00.377090738Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3da97f2b-8dc4-47f0-81ba-91f4190a2d02
            correlation ID: e08d62dc-1750-4dd8-bfbf-0bd47a4c0c8a
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0x39761e268740>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-dm68szdn2k7s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3da97f2b-8dc4-47f0-81ba-91f4190a2d02
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3da97f2b-8dc4-47f0-81ba-91f4190a2d02",
      "name": "3da97f2b-8dc4-47f0-81ba-91f4190a2d02",
      "status": "Failed",
      "startTime": "2026-10-22T02:26:51.272492265Z",
      "endTime": "2026-10-22T02:56:00.377090738Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3da97f2b-8dc4-47f0-81ba-91f4190a2d02
            correlation ID: e08d62dc-1750-4dd8-bfbf-0bd47a4c0c8a
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-4-20-kwjc5xlqs8tk/cluster-candidate-4-20-bcqtqw should provision
Unexpected error:
    <*fmt.wrapError | 0x361f110a43c0>: 
    failed to create HCP cluster cluster-candidate-4-20-bcqtqw: failed waiting for cluster="cluster-candidate-4-20-bcqtqw" in resourcegroup="rg-candidate-4-20-kwjc5xlqs8tk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9d6851f9-9f2d-451b-b4e5-4ae16700f297
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9d6851f9-9f2d-451b-b4e5-4ae16700f297",
      "name": "9d6851f9-9f2d-451b-b4e5-4ae16700f297",
      "status": "Failed",
      "startTime": "2026-10-22T02:21:15.911807316Z",
      "endTime": "2026-10-22T02:50:20.515370897Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, azure-cloud-controller-manager, csi-snapshot-controller-operator, olm-collect-profiles, cluster-version-operator, cluster-image-registry-operator, cluster-autoscaler, machine-approver, konnectivity-agent, cluster-policy-controller, control-plane-pki-operator, kube-scheduler, openshift-route-controller-manager, dns-operator, ingress-operator, cluster-node-tuning-operator, redhat-operators-catalog, certified-operators-catalog, kube-controller-manager, cluster-storage-operator, catalog-operator, olm-operator, packageserver, openshift-controller-manager, cluster-network-operator, redhat-marketplace-catalog, community-operators-catalog, openshift-apiserver; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9d6851f9-9f2d-451b-b4e5-4ae16700f297
            correlation ID: a2c7933e-26ff-48cd-92c5-66f72a0f2820
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-4-20-kwjc5xlqs8tk/cluster-candidate-4-20-bcqtqw should provision
Unexpected error:
    <*fmt.wrapError | 0x361f110a43c0>: 
    failed to create HCP cluster cluster-candidate-4-20-bcqtqw: failed waiting for cluster="cluster-candidate-4-20-bcqtqw" in resourcegroup="rg-candidate-4-20-kwjc5xlqs8tk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9d6851f9-9f2d-451b-b4e5-4ae16700f297
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9d6851f9-9f2d-451b-b4e5-4ae16700f297",
      "name": "9d6851f9-9f2d-451b-b4e5-4ae16700f297",
      "status": "Failed",
      "startTime": "2026-10-22T02:21:15.911807316Z",
      "endTime": "2026-10-22T02:50:20.515370897Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, azure-cloud-controller-manager, csi-snapshot-controller-operator, olm-collect-profiles, cluster-version-operator, cluster-image-registry-operator, cluster-autoscaler, machine-approver, konnectivity-agent, cluster-policy-controller, control-plane-pki-operator, kube-scheduler, openshift-route-controller-manager, dns-operator, ingress-operator, cluster-node-tuning-operator, redhat-operators-catalog, certified-operators-catalog, kube-controller-manager, cluster-storage-operator, catalog-operator, olm-operator, packageserver, openshift-controller-manager, cluster-network-operator, redhat-marketplace-catalog, community-operators-catalog, openshift-apiserver; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9d6851f9-9f2d-451b-b4e5-4ae16700f297
            correlation ID: a2c7933e-26ff-48cd-92c5-66f72a0f2820
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0x7938a410140>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-ql2jtgg2jsjl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/65c2874a-4e4d-4754-a6c4-c23ee1f34bb9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/65c2874a-4e4d-4754-a6c4-c23ee1f34bb9",
      "name": "65c2874a-4e4d-4754-a6c4-c23ee1f34bb9",
      "status": "Failed",
      "startTime": "2026-10-22T01:38:14.381676151Z",
      "endTime": "2026-10-22T02:07:16.666379289Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/65c2874a-4e4d-4754-a6c4-c23ee1f34bb9
            correlation ID: 3892a8ef-862d-414b-aadd-333496c0245e
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0x7938a410140>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-ql2jtgg2jsjl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/65c2874a-4e4d-4754-a6c4-c23ee1f34bb9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/65c2874a-4e4d-4754-a6c4-c23ee1f34bb9",
      "name": "65c2874a-4e4d-4754-a6c4-c23ee1f34bb9",
      "status": "Failed",
      "startTime": "2026-10-22T01:38:14.381676151Z",
      "endTime": "2026-10-22T02:07:16.666379289Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/65c2874a-4e4d-4754-a6c4-c23ee1f34bb9
            correlation ID: 3892a8ef-862d-414b-aadd-333496c0245e
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-apiserver]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x283eb71121c0>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-wqltj9p8d5tp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07e1caae-294a-4885-b57d-e584ce7036c1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07e1caae-294a-4885-b57d-e584ce7036c1",
      "name": "07e1caae-294a-4885-b57d-e584ce7036c1",
      "status": "Failed",
      "startTime": "2026-10-22T03:11:14.830364623Z",
      "endTime": "2026-10-22T03:40:15.05749374Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, kube-controller-manager, kube-scheduler, openshift-apiserver, cluster-storage-operator, ingress-operator, community-operators-catalog, cluster-version-operator, redhat-marketplace-catalog, certified-operators-catalog, azure-cloud-controller-manager, cluster-network-operator, cluster-image-registry-operator, konnectivity-agent, machine-approver, csi-snapshot-controller-operator, packageserver, cluster-policy-controller, dns-operator, catalog-operator, redhat-operators-catalog, cluster-autoscaler, olm-operator, control-plane-pki-operator, hosted-cluster-config-operator, openshift-controller-manager, cluster-node-tuning-operator, olm-collect-profiles; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07e1caae-294a-4885-b57d-e584ce7036c1
            correlation ID: 042010a3-104c-478c-8958-4456058069f0
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x283eb71121c0>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-wqltj9p8d5tp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07e1caae-294a-4885-b57d-e584ce7036c1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07e1caae-294a-4885-b57d-e584ce7036c1",
      "name": "07e1caae-294a-4885-b57d-e584ce7036c1",
      "status": "Failed",
      "startTime": "2026-10-22T03:11:14.830364623Z",
      "endTime": "2026-10-22T03:40:15.05749374Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, kube-controller-manager, kube-scheduler, openshift-apiserver, cluster-storage-operator, ingress-operator, community-operators-catalog, cluster-version-operator, redhat-marketplace-catalog, certified-operators-catalog, azure-cloud-controller-manager, cluster-network-operator, cluster-image-registry-operator, konnectivity-agent, machine-approver, csi-snapshot-controller-operator, packageserver, cluster-policy-controller, dns-operator, catalog-operator, redhat-operators-catalog, cluster-autoscaler, olm-operator, control-plane-pki-operator, hosted-cluster-config-operator, openshift-controller-manager, cluster-node-tuning-operator, olm-collect-profiles; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07e1caae-294a-4885-b57d-e584ce7036c1
            correlation ID: 042010a3-104c-478c-8958-4456058069f0
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0x144fc2ee5040>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-kfpbsf629cbt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b6385dea-fc39-4d5c-bf4a-4517f6ef3e1e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b6385dea-fc39-4d5c-bf4a-4517f6ef3e1e",
      "name": "b6385dea-fc39-4d5c-bf4a-4517f6ef3e1e",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:03.962666335Z",
      "endTime": "2026-10-22T02:03:06.792539952Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b6385dea-fc39-4d5c-bf4a-4517f6ef3e1e
            correlation ID: f2690118-3948-45a7-a4ae-1b177d998957
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0x144fc2ee5040>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-kfpbsf629cbt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b6385dea-fc39-4d5c-bf4a-4517f6ef3e1e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b6385dea-fc39-4d5c-bf4a-4517f6ef3e1e",
      "name": "b6385dea-fc39-4d5c-bf4a-4517f6ef3e1e",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:03.962666335Z",
      "endTime": "2026-10-22T02:03:06.792539952Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b6385dea-fc39-4d5c-bf4a-4517f6ef3e1e
            correlation ID: f2690118-3948-45a7-a4ae-1b177d998957
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server-proxy]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-nbxbs6
Unexpected error:
    <*fmt.wrapError | 0x24bdb9a64a20>: 
    failed to create HCP cluster np-noedge-nbxbs6: failed waiting for cluster="np-noedge-nbxbs6" in resourcegroup="rg-np-noedge-nbxbs6-kqkfkllsq598" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66bec196-bb1d-462f-9d3a-76228415a662
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66bec196-bb1d-462f-9d3a-76228415a662",
      "name": "66bec196-bb1d-462f-9d3a-76228415a662",
      "status": "Failed",
      "startTime": "2026-10-22T03:10:43.148572951Z",
      "endTime": "2026-10-22T03:39:45.079565863Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66bec196-bb1d-462f-9d3a-76228415a662
            correlation ID: 547b3712-e9f5-410b-87e2-dd2721dd720f
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-nbxbs6
Unexpected error:
    <*fmt.wrapError | 0x24bdb9a64a20>: 
    failed to create HCP cluster np-noedge-nbxbs6: failed waiting for cluster="np-noedge-nbxbs6" in resourcegroup="rg-np-noedge-nbxbs6-kqkfkllsq598" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66bec196-bb1d-462f-9d3a-76228415a662
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66bec196-bb1d-462f-9d3a-76228415a662",
      "name": "66bec196-bb1d-462f-9d3a-76228415a662",
      "status": "Failed",
      "startTime": "2026-10-22T03:10:43.148572951Z",
      "endTime": "2026-10-22T03:39:45.079565863Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66bec196-bb1d-462f-9d3a-76228415a662
            correlation ID: 547b3712-e9f5-410b-87e2-dd2721dd720f
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:86]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0x12e0378a8e0>: 
    failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-s6hb8822zv5x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b0fcfc1d-56f0-4a0e-be27-eb59d78d3391
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b0fcfc1d-56f0-4a0e-be27-eb59d78d3391",
      "name": "b0fcfc1d-56f0-4a0e-be27-eb59d78d3391",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:12.606084954Z",
      "endTime": "2026-10-22T02:03:16.732035531Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b0fcfc1d-56f0-4a0e-be27-eb59d78d3391
            correlation ID: d29a315a-5b37-4a91-af7b-1415cfe4e5ec
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:86]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0x12e0378a8e0>: 
    failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-s6hb8822zv5x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b0fcfc1d-56f0-4a0e-be27-eb59d78d3391
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b0fcfc1d-56f0-4a0e-be27-eb59d78d3391",
      "name": "b0fcfc1d-56f0-4a0e-be27-eb59d78d3391",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:12.606084954Z",
      "endTime": "2026-10-22T02:03:16.732035531Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b0fcfc1d-56f0-4a0e-be27-eb59d78d3391
            correlation ID: d29a315a-5b37-4a91-af7b-1415cfe4e5ec
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xd879a7a0b60>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-7ht4n2l8tbfm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94676dae-e02f-4af7-9503-8526a46b136a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94676dae-e02f-4af7-9503-8526a46b136a",
      "name": "94676dae-e02f-4af7-9503-8526a46b136a",
      "status": "Failed",
      "startTime": "2026-10-22T01:33:54.675375561Z",
      "endTime": "2026-10-22T02:02:56.694987866Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94676dae-e02f-4af7-9503-8526a46b136a
            correlation ID: eb262c16-35c1-4306-89d7-1ab7a4a6f8e5
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xd879a7a0b60>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-7ht4n2l8tbfm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94676dae-e02f-4af7-9503-8526a46b136a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94676dae-e02f-4af7-9503-8526a46b136a",
      "name": "94676dae-e02f-4af7-9503-8526a46b136a",
      "status": "Failed",
      "startTime": "2026-10-22T01:33:54.675375561Z",
      "endTime": "2026-10-22T02:02:56.694987866Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94676dae-e02f-4af7-9503-8526a46b136a
            correlation ID: eb262c16-35c1-4306-89d7-1ab7a4a6f8e5
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0x3e201b18a700>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-w4bh6t74lrm8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ffa3393-14e6-485e-a06e-d34ad16e45af
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ffa3393-14e6-485e-a06e-d34ad16e45af",
      "name": "3ffa3393-14e6-485e-a06e-d34ad16e45af",
      "status": "Failed",
      "startTime": "2026-10-22T01:33:57.248605332Z",
      "endTime": "2026-10-22T02:03:06.666838162Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, cluster-policy-controller, olm-operator, openshift-route-controller-manager, cluster-storage-operator, packageserver, redhat-marketplace-catalog, community-operators-catalog, cluster-network-operator, dns-operator, ingress-operator, catalog-operator, certified-operators-catalog, redhat-operators-catalog, hosted-cluster-config-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, openshift-controller-manager, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ffa3393-14e6-485e-a06e-d34ad16e45af
            correlation ID: d4851c32-d0b7-4ef1-9333-ecb958420c6c
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0x3e201b18a700>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-w4bh6t74lrm8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ffa3393-14e6-485e-a06e-d34ad16e45af
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ffa3393-14e6-485e-a06e-d34ad16e45af",
      "name": "3ffa3393-14e6-485e-a06e-d34ad16e45af",
      "status": "Failed",
      "startTime": "2026-10-22T01:33:57.248605332Z",
      "endTime": "2026-10-22T02:03:06.666838162Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, cluster-policy-controller, olm-operator, openshift-route-controller-manager, cluster-storage-operator, packageserver, redhat-marketplace-catalog, community-operators-catalog, cluster-network-operator, dns-operator, ingress-operator, catalog-operator, certified-operators-catalog, redhat-operators-catalog, hosted-cluster-config-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, openshift-controller-manager, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ffa3393-14e6-485e-a06e-d34ad16e45af
            correlation ID: d4851c32-d0b7-4ef1-9333-ecb958420c6c
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-21-xzrpbt" with version 4.21.38 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x913e9d26a60>: 
    failed to create HCP cluster cluster-zstream-4-21-xzrpbt: failed waiting for cluster="cluster-zstream-4-21-xzrpbt" in resourcegroup="rg-zstream-upgrade-4-21-xgpgfghfg7wk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d76aa69d-45fb-439f-889e-97ea3ff34f99
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d76aa69d-45fb-439f-889e-97ea3ff34f99",
      "name": "d76aa69d-45fb-439f-889e-97ea3ff34f99",
      "status": "Failed",
      "startTime": "2026-10-22T02:22:56.334901591Z",
      "endTime": "2026-10-22T02:52:00.375914793Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, dns-operator, redhat-operators-catalog, certified-operators-catalog, control-plane-pki-operator, cluster-version-operator, openshift-route-controller-manager, azure-cloud-controller-manager, redhat-marketplace-catalog, cluster-autoscaler, hosted-cluster-config-operator, olm-operator, packageserver, ingress-operator, cluster-image-registry-operator, machine-approver, cluster-policy-controller, cluster-node-tuning-operator, konnectivity-agent, kube-controller-manager, openshift-controller-manager, cluster-network-operator, catalog-operator, community-operators-catalog, kube-scheduler, openshift-apiserver, csi-snapshot-controller-operator, olm-collect-profiles; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d76aa69d-45fb-439f-889e-97ea3ff34f99
            correlation ID: b1fc9a13-2fc1-4373-b2a1-ce2064832e2e
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-21-xzrpbt" with version 4.21.38 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x913e9d26a60>: 
    failed to create HCP cluster cluster-zstream-4-21-xzrpbt: failed waiting for cluster="cluster-zstream-4-21-xzrpbt" in resourcegroup="rg-zstream-upgrade-4-21-xgpgfghfg7wk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d76aa69d-45fb-439f-889e-97ea3ff34f99
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d76aa69d-45fb-439f-889e-97ea3ff34f99",
      "name": "d76aa69d-45fb-439f-889e-97ea3ff34f99",
      "status": "Failed",
      "startTime": "2026-10-22T02:22:56.334901591Z",
      "endTime": "2026-10-22T02:52:00.375914793Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, dns-operator, redhat-operators-catalog, certified-operators-catalog, control-plane-pki-operator, cluster-version-operator, openshift-route-controller-manager, azure-cloud-controller-manager, redhat-marketplace-catalog, cluster-autoscaler, hosted-cluster-config-operator, olm-operator, packageserver, ingress-operator, cluster-image-registry-operator, machine-approver, cluster-policy-controller, cluster-node-tuning-operator, konnectivity-agent, kube-controller-manager, openshift-controller-manager, cluster-network-operator, catalog-operator, community-operators-catalog, kube-scheduler, openshift-apiserver, csi-snapshot-controller-operator, olm-collect-profiles; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d76aa69d-45fb-439f-889e-97ea3ff34f99
            correlation ID: b1fc9a13-2fc1-4373-b2a1-ce2064832e2e
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:582]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x115fb34512c0>: 
    failed to create HCP cluster np-downgrade-42l48k: failed waiting for cluster="np-downgrade-42l48k" in resourcegroup="rg-np-downgrade-42l48k-5zqzd6q4wnms" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/05348ada-50c8-472d-8c3c-a7bf34489fb8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/05348ada-50c8-472d-8c3c-a7bf34489fb8",
      "name": "05348ada-50c8-472d-8c3c-a7bf34489fb8",
      "status": "Failed",
      "startTime": "2026-10-22T03:11:02.734882718Z",
      "endTime": "2026-10-22T03:40:05.149727722Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, cluster-storage-operator, community-operators-catalog, machine-approver, openshift-route-controller-manager, cluster-policy-controller, cluster-image-registry-operator, packageserver, control-plane-pki-operator, dns-operator, openshift-apiserver, cluster-node-tuning-operator, kube-controller-manager, hosted-cluster-config-operator, cluster-network-operator, konnectivity-agent, olm-collect-profiles, azure-cloud-controller-manager, csi-snapshot-controller-operator, olm-operator, redhat-operators-catalog, certified-operators-catalog, cluster-autoscaler, ingress-operator, redhat-marketplace-catalog, catalog-operator, kube-scheduler, cluster-version-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/05348ada-50c8-472d-8c3c-a7bf34489fb8
            correlation ID: 513d2a7a-52d7-4655-b715-b68c051bb6dc
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:582]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x115fb34512c0>: 
    failed to create HCP cluster np-downgrade-42l48k: failed waiting for cluster="np-downgrade-42l48k" in resourcegroup="rg-np-downgrade-42l48k-5zqzd6q4wnms" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/05348ada-50c8-472d-8c3c-a7bf34489fb8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/05348ada-50c8-472d-8c3c-a7bf34489fb8",
      "name": "05348ada-50c8-472d-8c3c-a7bf34489fb8",
      "status": "Failed",
      "startTime": "2026-10-22T03:11:02.734882718Z",
      "endTime": "2026-10-22T03:40:05.149727722Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, cluster-storage-operator, community-operators-catalog, machine-approver, openshift-route-controller-manager, cluster-policy-controller, cluster-image-registry-operator, packageserver, control-plane-pki-operator, dns-operator, openshift-apiserver, cluster-node-tuning-operator, kube-controller-manager, hosted-cluster-config-operator, cluster-network-operator, konnectivity-agent, olm-collect-profiles, azure-cloud-controller-manager, csi-snapshot-controller-operator, olm-operator, redhat-operators-catalog, certified-operators-catalog, cluster-autoscaler, ingress-operator, redhat-marketplace-catalog, catalog-operator, kube-scheduler, cluster-version-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/05348ada-50c8-472d-8c3c-a7bf34489fb8
            correlation ID: 513d2a7a-52d7-4655-b715-b68c051bb6dc
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0x331e8e5a0000>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-vj9q24f28wn9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d012ea11-280a-4508-8665-b98bf45a8243
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d012ea11-280a-4508-8665-b98bf45a8243",
      "name": "d012ea11-280a-4508-8665-b98bf45a8243",
      "status": "Failed",
      "startTime": "2026-10-22T02:25:30.550148133Z",
      "endTime": "2026-10-22T02:54:30.366190637Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d012ea11-280a-4508-8665-b98bf45a8243
            correlation ID: c24dd16b-dee6-4cf8-8524-5a4ca58ee6d0
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0x331e8e5a0000>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-vj9q24f28wn9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d012ea11-280a-4508-8665-b98bf45a8243
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d012ea11-280a-4508-8665-b98bf45a8243",
      "name": "d012ea11-280a-4508-8665-b98bf45a8243",
      "status": "Failed",
      "startTime": "2026-10-22T02:25:30.550148133Z",
      "endTime": "2026-10-22T02:54:30.366190637Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d012ea11-280a-4508-8665-b98bf45a8243
            correlation ID: c24dd16b-dee6-4cf8-8524-5a4ca58ee6d0
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0x3ad1958370e0>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-6z47gdxtfxk2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/88d7d257-b7b4-4fd6-968b-d958c1947661
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/88d7d257-b7b4-4fd6-968b-d958c1947661",
      "name": "88d7d257-b7b4-4fd6-968b-d958c1947661",
      "status": "Failed",
      "startTime": "2026-10-22T03:09:11.887412502Z",
      "endTime": "2026-10-22T03:38:15.157786729Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, certified-operators-catalog, dns-operator, catalog-operator, cluster-version-operator, cluster-network-operator, cluster-node-tuning-operator, redhat-operators-catalog, community-operators-catalog, olm-operator, packageserver, openshift-route-controller-manager, kube-controller-manager, kube-scheduler, csi-snapshot-controller-operator, openshift-controller-manager, hosted-cluster-config-operator, redhat-marketplace-catalog, control-plane-pki-operator, ingress-operator, konnectivity-agent, machine-approver, cluster-policy-controller, azure-cloud-controller-manager, cluster-storage-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/88d7d257-b7b4-4fd6-968b-d958c1947661
            correlation ID: 20139e98-2ffa-42ab-b589-fa9d706c6e08
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0x3ad1958370e0>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-6z47gdxtfxk2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/88d7d257-b7b4-4fd6-968b-d958c1947661
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/88d7d257-b7b4-4fd6-968b-d958c1947661",
      "name": "88d7d257-b7b4-4fd6-968b-d958c1947661",
      "status": "Failed",
      "startTime": "2026-10-22T03:09:11.887412502Z",
      "endTime": "2026-10-22T03:38:15.157786729Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, certified-operators-catalog, dns-operator, catalog-operator, cluster-version-operator, cluster-network-operator, cluster-node-tuning-operator, redhat-operators-catalog, community-operators-catalog, olm-operator, packageserver, openshift-route-controller-manager, kube-controller-manager, kube-scheduler, csi-snapshot-controller-operator, openshift-controller-manager, hosted-cluster-config-operator, redhat-marketplace-catalog, control-plane-pki-operator, ingress-operator, konnectivity-agent, machine-approver, cluster-policy-controller, azure-cloud-controller-manager, cluster-storage-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/88d7d257-b7b4-4fd6-968b-d958c1947661
            correlation ID: 20139e98-2ffa-42ab-b589-fa9d706c6e08
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-nnvzdg with version 4.21.39
Unexpected error:
    <*fmt.wrapError | 0x2bd5ec4c4840>: 
    failed to create HCP cluster np-version-upgrade-cluster-nnvzdg: failed waiting for cluster="np-version-upgrade-cluster-nnvzdg" in resourcegroup="rg-np-version-upgrade-nnvzdg-mqmfl8g2qxgk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/928f3c23-211a-4f82-842c-98d26e84d456
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/928f3c23-211a-4f82-842c-98d26e84d456",
      "name": "928f3c23-211a-4f82-842c-98d26e84d456",
      "status": "Failed",
      "startTime": "2026-10-22T03:09:47.131790254Z",
      "endTime": "2026-10-22T03:38:55.090650407Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, cluster-policy-controller, azure-cloud-controller-manager, csi-snapshot-controller-operator, certified-operators-catalog, machine-approver, hosted-cluster-config-operator, cluster-image-registry-operator, redhat-marketplace-catalog, community-operators-catalog, catalog-operator, kube-scheduler, openshift-apiserver, olm-collect-profiles, ingress-operator, cluster-version-operator, kube-controller-manager, cluster-storage-operator, packageserver, olm-operator, cluster-autoscaler, dns-operator, openshift-controller-manager, control-plane-pki-operator, cluster-network-operator, redhat-operators-catalog, cluster-node-tuning-operator, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/928f3c23-211a-4f82-842c-98d26e84d456
            correlation ID: 0bf299e5-3230-4f67-8cf1-6944f93fcb2e
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-nnvzdg with version 4.21.39
Unexpected error:
    <*fmt.wrapError | 0x2bd5ec4c4840>: 
    failed to create HCP cluster np-version-upgrade-cluster-nnvzdg: failed waiting for cluster="np-version-upgrade-cluster-nnvzdg" in resourcegroup="rg-np-version-upgrade-nnvzdg-mqmfl8g2qxgk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/928f3c23-211a-4f82-842c-98d26e84d456
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/928f3c23-211a-4f82-842c-98d26e84d456",
      "name": "928f3c23-211a-4f82-842c-98d26e84d456",
      "status": "Failed",
      "startTime": "2026-10-22T03:09:47.131790254Z",
      "endTime": "2026-10-22T03:38:55.090650407Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, cluster-policy-controller, azure-cloud-controller-manager, csi-snapshot-controller-operator, certified-operators-catalog, machine-approver, hosted-cluster-config-operator, cluster-image-registry-operator, redhat-marketplace-catalog, community-operators-catalog, catalog-operator, kube-scheduler, openshift-apiserver, olm-collect-profiles, ingress-operator, cluster-version-operator, kube-controller-manager, cluster-storage-operator, packageserver, olm-operator, cluster-autoscaler, dns-operator, openshift-controller-manager, control-plane-pki-operator, cluster-network-operator, redhat-operators-catalog, cluster-node-tuning-operator, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/928f3c23-211a-4f82-842c-98d26e84d456
            correlation ID: 0bf299e5-3230-4f67-8cf1-6944f93fcb2e
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:114]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0x13f4cc33f120>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-vwdnmjj8rtd5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/859603f2-b753-4e20-87c5-5cd7720f1d8e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/859603f2-b753-4e20-87c5-5cd7720f1d8e",
      "name": "859603f2-b753-4e20-87c5-5cd7720f1d8e",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:07.162285549Z",
      "endTime": "2026-10-22T02:03:16.736233168Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/859603f2-b753-4e20-87c5-5cd7720f1d8e
            correlation ID: 0121aed1-37e4-4ef9-9b80-ddb831aa669a
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:114]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0x13f4cc33f120>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-vwdnmjj8rtd5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/859603f2-b753-4e20-87c5-5cd7720f1d8e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/859603f2-b753-4e20-87c5-5cd7720f1d8e",
      "name": "859603f2-b753-4e20-87c5-5cd7720f1d8e",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:07.162285549Z",
      "endTime": "2026-10-22T02:03:16.736233168Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/859603f2-b753-4e20-87c5-5cd7720f1d8e
            correlation ID: 0121aed1-37e4-4ef9-9b80-ddb831aa669a
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, konnectivity-agent, kube-controller-manager, kube-scheduler, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-vwsld8 with version 4.20.43
Unexpected error:
    <*fmt.wrapError | 0xc02210a0420>: 
    failed to create HCP cluster np-version-upgrade-cluster-vwsld8: failed waiting for cluster="np-version-upgrade-cluster-vwsld8" in resourcegroup="rg-np-version-upgrade-vwsld8-b78v68n9cqj2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7061ec27-2552-401b-a8b5-5bcf7cc27448
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7061ec27-2552-401b-a8b5-5bcf7cc27448",
      "name": "7061ec27-2552-401b-a8b5-5bcf7cc27448",
      "status": "Failed",
      "startTime": "2026-10-22T03:10:01.822651936Z",
      "endTime": "2026-10-22T03:39:05.079618983Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, hosted-cluster-config-operator, cluster-image-registry-operator, catalog-operator, cluster-storage-operator, csi-snapshot-controller-operator, community-operators-catalog, cluster-node-tuning-operator, ingress-operator, redhat-marketplace-catalog, openshift-controller-manager, packageserver, cluster-policy-controller, dns-operator, olm-operator, cluster-network-operator, redhat-operators-catalog, certified-operators-catalog, openshift-route-controller-manager, konnectivity-agent, kube-controller-manager; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7061ec27-2552-401b-a8b5-5bcf7cc27448
            correlation ID: 8d4a80f0-34d4-471d-b30b-2c4ee315636d
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-vwsld8 with version 4.20.43
Unexpected error:
    <*fmt.wrapError | 0xc02210a0420>: 
    failed to create HCP cluster np-version-upgrade-cluster-vwsld8: failed waiting for cluster="np-version-upgrade-cluster-vwsld8" in resourcegroup="rg-np-version-upgrade-vwsld8-b78v68n9cqj2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7061ec27-2552-401b-a8b5-5bcf7cc27448
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7061ec27-2552-401b-a8b5-5bcf7cc27448",
      "name": "7061ec27-2552-401b-a8b5-5bcf7cc27448",
      "status": "Failed",
      "startTime": "2026-10-22T03:10:01.822651936Z",
      "endTime": "2026-10-22T03:39:05.079618983Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, hosted-cluster-config-operator, cluster-image-registry-operator, catalog-operator, cluster-storage-operator, csi-snapshot-controller-operator, community-operators-catalog, cluster-node-tuning-operator, ingress-operator, redhat-marketplace-catalog, openshift-controller-manager, packageserver, cluster-policy-controller, dns-operator, olm-operator, cluster-network-operator, redhat-operators-catalog, certified-operators-catalog, openshift-route-controller-manager, konnectivity-agent, kube-controller-manager; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7061ec27-2552-401b-a8b5-5bcf7cc27448
            correlation ID: 8d4a80f0-34d4-471d-b30b-2c4ee315636d
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0x85472092ba0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-d4m4wrpd9cv2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ed19878-5858-4619-b204-f2d81fa433c2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ed19878-5858-4619-b204-f2d81fa433c2",
      "name": "8ed19878-5858-4619-b204-f2d81fa433c2",
      "status": "Failed",
      "startTime": "2026-10-22T01:35:56.765676016Z",
      "endTime": "2026-10-22T02:04:56.661581883Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ed19878-5858-4619-b204-f2d81fa433c2
            correlation ID: 6f084968-c4fe-49c7-b4c8-bb07b85c93c8
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0x85472092ba0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-d4m4wrpd9cv2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ed19878-5858-4619-b204-f2d81fa433c2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ed19878-5858-4619-b204-f2d81fa433c2",
      "name": "8ed19878-5858-4619-b204-f2d81fa433c2",
      "status": "Failed",
      "startTime": "2026-10-22T01:35:56.765676016Z",
      "endTime": "2026-10-22T02:04:56.661581883Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ed19878-5858-4619-b204-f2d81fa433c2
            correlation ID: 6f084968-c4fe-49c7-b4c8-bb07b85c93c8
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0x19f84045ab20>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-pbcvbtl256h6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2754f526-7d96-46a2-b85d-5f23a12e42c5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2754f526-7d96-46a2-b85d-5f23a12e42c5",
      "name": "2754f526-7d96-46a2-b85d-5f23a12e42c5",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:31.419619024Z",
      "endTime": "2026-10-22T02:03:36.794767033Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, kube-controller-manager, packageserver, machine-approver, cluster-policy-controller, konnectivity-agent, certified-operators-catalog, control-plane-pki-operator, azure-cloud-controller-manager, olm-operator, olm-collect-profiles, redhat-marketplace-catalog, community-operators-catalog, cluster-autoscaler, openshift-controller-manager, openshift-route-controller-manager, kube-scheduler, hosted-cluster-config-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, openshift-apiserver, cluster-version-operator, cluster-network-operator, dns-operator, redhat-operators-catalog, cluster-storage-operator, cluster-node-tuning-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2754f526-7d96-46a2-b85d-5f23a12e42c5
            correlation ID: 65497397-2503-43f4-9c67-ed34b0edfa0b
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0x19f84045ab20>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-pbcvbtl256h6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2754f526-7d96-46a2-b85d-5f23a12e42c5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2754f526-7d96-46a2-b85d-5f23a12e42c5",
      "name": "2754f526-7d96-46a2-b85d-5f23a12e42c5",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:31.419619024Z",
      "endTime": "2026-10-22T02:03:36.794767033Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, kube-controller-manager, packageserver, machine-approver, cluster-policy-controller, konnectivity-agent, certified-operators-catalog, control-plane-pki-operator, azure-cloud-controller-manager, olm-operator, olm-collect-profiles, redhat-marketplace-catalog, community-operators-catalog, cluster-autoscaler, openshift-controller-manager, openshift-route-controller-manager, kube-scheduler, hosted-cluster-config-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, openshift-apiserver, cluster-version-operator, cluster-network-operator, dns-operator, redhat-operators-catalog, cluster-storage-operator, cluster-node-tuning-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2754f526-7d96-46a2-b85d-5f23a12e42c5
            correlation ID: 65497397-2503-43f4-9c67-ed34b0edfa0b
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-controller-manager]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should be able to delete an HCP cluster whose managed identities were deleted first · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0x28c6381f07c0>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed waiting for cluster="missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-plwcnbx7hnw5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5f96f74a-1c0c-4295-a2c6-9dce177af3f9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5f96f74a-1c0c-4295-a2c6-9dce177af3f9",
      "name": "5f96f74a-1c0c-4295-a2c6-9dce177af3f9",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:31.83333937Z",
      "endTime": "2026-10-22T02:03:36.684949695Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 24m44s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5f96f74a-1c0c-4295-a2c6-9dce177af3f9
            correlation ID: aad1d81c-32be-4413-84a9-7ac0b5dcc1a8
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0x28c6381f07c0>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed waiting for cluster="missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-plwcnbx7hnw5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5f96f74a-1c0c-4295-a2c6-9dce177af3f9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5f96f74a-1c0c-4295-a2c6-9dce177af3f9",
      "name": "5f96f74a-1c0c-4295-a2c6-9dce177af3f9",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:31.83333937Z",
      "endTime": "2026-10-22T02:03:36.684949695Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 24m44s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5f96f74a-1c0c-4295-a2c6-9dce177af3f9
            correlation ID: aad1d81c-32be-4413-84a9-7ac0b5dcc1a8
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, cluster-api, ignition-server]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-bbjx5c" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0x2052152a2720>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-bbjx5c: failed waiting for cluster="cp-ystream-upgrade-4-23-bbjx5c" in resourcegroup="rg-cp-ystream-upgrade-4-23-56g2lp75qrkf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/02ef6437-1d48-43c3-ad2d-b3b8f18f1b53
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/02ef6437-1d48-43c3-ad2d-b3b8f18f1b53",
      "name": "02ef6437-1d48-43c3-ad2d-b3b8f18f1b53",
      "status": "Failed",
      "startTime": "2026-10-22T02:23:39.532402776Z",
      "endTime": "2026-10-22T02:52:40.523543888Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, cluster-api deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/02ef6437-1d48-43c3-ad2d-b3b8f18f1b53
            correlation ID: 43512322-23b0-4759-b738-201cec5eb6f3
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-bbjx5c" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0x2052152a2720>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-bbjx5c: failed waiting for cluster="cp-ystream-upgrade-4-23-bbjx5c" in resourcegroup="rg-cp-ystream-upgrade-4-23-56g2lp75qrkf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/02ef6437-1d48-43c3-ad2d-b3b8f18f1b53
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/02ef6437-1d48-43c3-ad2d-b3b8f18f1b53",
      "name": "02ef6437-1d48-43c3-ad2d-b3b8f18f1b53",
      "status": "Failed",
      "startTime": "2026-10-22T02:23:39.532402776Z",
      "endTime": "2026-10-22T02:52:40.523543888Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, cluster-api deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/02ef6437-1d48-43c3-ad2d-b3b8f18f1b53
            correlation ID: 43512322-23b0-4759-b738-201cec5eb6f3
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0x353e84400c40>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-mmwcq8j6x77j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/045880bb-951e-46ab-b5a0-e93973ee8401
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/045880bb-951e-46ab-b5a0-e93973ee8401",
      "name": "045880bb-951e-46ab-b5a0-e93973ee8401",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:06.474310765Z",
      "endTime": "2026-10-22T02:03:06.762878168Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-node-tuning-operator, cluster-policy-controller, dns-operator, redhat-operators-catalog, redhat-marketplace-catalog, community-operators-catalog, control-plane-pki-operator, kube-scheduler, openshift-controller-manager, cluster-network-operator, cluster-image-registry-operator, cluster-version-operator, ingress-operator, packageserver, hosted-cluster-config-operator, catalog-operator, cluster-storage-operator, csi-snapshot-controller-operator, konnectivity-agent, machine-approver, kube-controller-manager, openshift-route-controller-manager, azure-cloud-controller-manager, olm-operator, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, capi-provider deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/045880bb-951e-46ab-b5a0-e93973ee8401
            correlation ID: 1f9e7a18-e51b-4413-80b7-a809b5d0f849
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0x353e84400c40>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-mmwcq8j6x77j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/045880bb-951e-46ab-b5a0-e93973ee8401
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/045880bb-951e-46ab-b5a0-e93973ee8401",
      "name": "045880bb-951e-46ab-b5a0-e93973ee8401",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:06.474310765Z",
      "endTime": "2026-10-22T02:03:06.762878168Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-node-tuning-operator, cluster-policy-controller, dns-operator, redhat-operators-catalog, redhat-marketplace-catalog, community-operators-catalog, control-plane-pki-operator, kube-scheduler, openshift-controller-manager, cluster-network-operator, cluster-image-registry-operator, cluster-version-operator, ingress-operator, packageserver, hosted-cluster-config-operator, catalog-operator, cluster-storage-operator, csi-snapshot-controller-operator, konnectivity-agent, machine-approver, kube-controller-manager, openshift-route-controller-manager, azure-cloud-controller-manager, olm-operator, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, capi-provider deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/045880bb-951e-46ab-b5a0-e93973ee8401
            correlation ID: 1f9e7a18-e51b-4413-80b7-a809b5d0f849
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-23-tvqgcd" with version 4.23.0-ec.0 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x3a1b74a0c6e0>: 
    failed to create HCP cluster cluster-zstream-4-23-tvqgcd: failed waiting for cluster="cluster-zstream-4-23-tvqgcd" in resourcegroup="rg-zstream-upgrade-4-23-z4qsqrdw854d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bdede344-c320-476c-93e8-c3141c6d343a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bdede344-c320-476c-93e8-c3141c6d343a",
      "name": "bdede344-c320-476c-93e8-c3141c6d343a",
      "status": "Failed",
      "startTime": "2026-10-22T02:22:59.55049493Z",
      "endTime": "2026-10-22T02:52:00.596740917Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, hosted-cluster-config-operator, cluster-node-tuning-operator, cluster-version-operator, openshift-controller-manager, cluster-network-operator, openshift-route-controller-manager, cluster-storage-operator, ingress-operator, csi-snapshot-controller-operator, redhat-operators-catalog, certified-operators-catalog, dns-operator, community-operators-catalog, olm-operator, cluster-image-registry-operator, catalog-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bdede344-c320-476c-93e8-c3141c6d343a
            correlation ID: 946c8b3d-71c1-46d3-ad96-760f309b6eb8
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-23-tvqgcd" with version 4.23.0-ec.0 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x3a1b74a0c6e0>: 
    failed to create HCP cluster cluster-zstream-4-23-tvqgcd: failed waiting for cluster="cluster-zstream-4-23-tvqgcd" in resourcegroup="rg-zstream-upgrade-4-23-z4qsqrdw854d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bdede344-c320-476c-93e8-c3141c6d343a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bdede344-c320-476c-93e8-c3141c6d343a",
      "name": "bdede344-c320-476c-93e8-c3141c6d343a",
      "status": "Failed",
      "startTime": "2026-10-22T02:22:59.55049493Z",
      "endTime": "2026-10-22T02:52:00.596740917Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, hosted-cluster-config-operator, cluster-node-tuning-operator, cluster-version-operator, openshift-controller-manager, cluster-network-operator, openshift-route-controller-manager, cluster-storage-operator, ingress-operator, csi-snapshot-controller-operator, redhat-operators-catalog, certified-operators-catalog, dns-operator, community-operators-catalog, olm-operator, cluster-image-registry-operator, catalog-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bdede344-c320-476c-93e8-c3141c6d343a
            correlation ID: 946c8b3d-71c1-46d3-ad96-760f309b6eb8
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0x327c31abadc0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-vzj54nlw5csp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9fa1671-b5d8-42db-9ac9-d89b520cbbf9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9fa1671-b5d8-42db-9ac9-d89b520cbbf9",
      "name": "f9fa1671-b5d8-42db-9ac9-d89b520cbbf9",
      "status": "Failed",
      "startTime": "2026-10-22T01:35:36.727726764Z",
      "endTime": "2026-10-22T02:04:36.824596885Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9fa1671-b5d8-42db-9ac9-d89b520cbbf9
            correlation ID: ba4affaa-364d-40fe-b039-d27e78c0a5e0
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0x327c31abadc0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-vzj54nlw5csp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9fa1671-b5d8-42db-9ac9-d89b520cbbf9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9fa1671-b5d8-42db-9ac9-d89b520cbbf9",
      "name": "f9fa1671-b5d8-42db-9ac9-d89b520cbbf9",
      "status": "Failed",
      "startTime": "2026-10-22T01:35:36.727726764Z",
      "endTime": "2026-10-22T02:04:36.824596885Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9fa1671-b5d8-42db-9ac9-d89b520cbbf9
            correlation ID: ba4affaa-364d-40fe-b039-d27e78c0a5e0
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xbfee40279a0>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-fx2f2r72cdzb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/157c737e-923c-42ef-b1d7-edf1ae3eb633
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/157c737e-923c-42ef-b1d7-edf1ae3eb633",
      "name": "157c737e-923c-42ef-b1d7-edf1ae3eb633",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:17.250629196Z",
      "endTime": "2026-10-22T02:03:26.781048315Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: control-plane-pki-operator, cluster-storage-operator, cluster-node-tuning-operator, cluster-image-registry-operator, openshift-controller-manager, hosted-cluster-config-operator, azure-cloud-controller-manager, catalog-operator, redhat-operators-catalog, redhat-marketplace-catalog, cluster-version-operator, dns-operator, ingress-operator, packageserver, machine-approver, openshift-apiserver, csi-snapshot-controller-operator, konnectivity-agent, certified-operators-catalog, cluster-policy-controller, cluster-network-operator, olm-collect-profiles, cluster-autoscaler, kube-scheduler, openshift-route-controller-manager, olm-operator, community-operators-catalog, kube-controller-manager; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/157c737e-923c-42ef-b1d7-edf1ae3eb633
            correlation ID: e8e05aa0-c13d-42e2-915a-25e9e6211e06
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xbfee40279a0>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-fx2f2r72cdzb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/157c737e-923c-42ef-b1d7-edf1ae3eb633
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/157c737e-923c-42ef-b1d7-edf1ae3eb633",
      "name": "157c737e-923c-42ef-b1d7-edf1ae3eb633",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:17.250629196Z",
      "endTime": "2026-10-22T02:03:26.781048315Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: control-plane-pki-operator, cluster-storage-operator, cluster-node-tuning-operator, cluster-image-registry-operator, openshift-controller-manager, hosted-cluster-config-operator, azure-cloud-controller-manager, catalog-operator, redhat-operators-catalog, redhat-marketplace-catalog, cluster-version-operator, dns-operator, ingress-operator, packageserver, machine-approver, openshift-apiserver, csi-snapshot-controller-operator, konnectivity-agent, certified-operators-catalog, cluster-policy-controller, cluster-network-operator, olm-collect-profiles, cluster-autoscaler, kube-scheduler, openshift-route-controller-manager, olm-operator, community-operators-catalog, kube-controller-manager; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/157c737e-923c-42ef-b1d7-edf1ae3eb633
            correlation ID: e8e05aa0-c13d-42e2-915a-25e9e6211e06
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-20-zj4mcv" with version 4.20.42 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x336a41a7c8a0>: 
    failed to create HCP cluster cluster-zstream-4-20-zj4mcv: failed waiting for cluster="cluster-zstream-4-20-zj4mcv" in resourcegroup="rg-zstream-upgrade-4-20-2z67w2qcbxs6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/03c686a3-8b51-4273-9c18-ce6884c5a033
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/03c686a3-8b51-4273-9c18-ce6884c5a033",
      "name": "03c686a3-8b51-4273-9c18-ce6884c5a033",
      "status": "Failed",
      "startTime": "2026-10-22T02:22:44.492283349Z",
      "endTime": "2026-10-22T02:51:50.526771089Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: certified-operators-catalog, cluster-storage-operator, cluster-image-registry-operator, catalog-operator, openshift-controller-manager, cluster-policy-controller, cluster-network-operator, dns-operator, redhat-operators-catalog, cluster-version-operator, hosted-cluster-config-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, redhat-marketplace-catalog, openshift-route-controller-manager, ingress-operator, olm-operator, packageserver, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/03c686a3-8b51-4273-9c18-ce6884c5a033
            correlation ID: 5cab7c5c-8f8d-4763-8268-cddbff9d06f5
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-20-zj4mcv" with version 4.20.42 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x336a41a7c8a0>: 
    failed to create HCP cluster cluster-zstream-4-20-zj4mcv: failed waiting for cluster="cluster-zstream-4-20-zj4mcv" in resourcegroup="rg-zstream-upgrade-4-20-2z67w2qcbxs6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/03c686a3-8b51-4273-9c18-ce6884c5a033
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/03c686a3-8b51-4273-9c18-ce6884c5a033",
      "name": "03c686a3-8b51-4273-9c18-ce6884c5a033",
      "status": "Failed",
      "startTime": "2026-10-22T02:22:44.492283349Z",
      "endTime": "2026-10-22T02:51:50.526771089Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: certified-operators-catalog, cluster-storage-operator, cluster-image-registry-operator, catalog-operator, openshift-controller-manager, cluster-policy-controller, cluster-network-operator, dns-operator, redhat-operators-catalog, cluster-version-operator, hosted-cluster-config-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, redhat-marketplace-catalog, openshift-route-controller-manager, ingress-operator, olm-operator, packageserver, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/03c686a3-8b51-4273-9c18-ce6884c5a033
            correlation ID: 5cab7c5c-8f8d-4763-8268-cddbff9d06f5
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, konnectivity-agent, kube-controller-manager, kube-scheduler, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-hlfqz4 with version 4.21.39
Unexpected error:
    <*fmt.wrapError | 0x26fe4f09e7e0>: 
    failed to create HCP cluster np-version-upgrade-cluster-hlfqz4: failed waiting for cluster="np-version-upgrade-cluster-hlfqz4" in resourcegroup="rg-np-version-upgrade-hlfqz4-hxtt44szpfn8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9b1bdec9-cda2-4c03-8af1-a62744ae5bac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9b1bdec9-cda2-4c03-8af1-a62744ae5bac",
      "name": "9b1bdec9-cda2-4c03-8af1-a62744ae5bac",
      "status": "Failed",
      "startTime": "2026-10-22T03:09:57.381547287Z",
      "endTime": "2026-10-22T03:39:05.056394482Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: konnectivity-agent, redhat-operators-catalog, catalog-operator, cluster-version-operator, cluster-network-operator, dns-operator, csi-snapshot-controller-operator, redhat-marketplace-catalog, cluster-policy-controller, cluster-storage-operator, openshift-route-controller-manager, kube-controller-manager, ingress-operator, packageserver, olm-operator, kube-scheduler, hosted-cluster-config-operator, openshift-controller-manager, cluster-node-tuning-operator, cluster-image-registry-operator, community-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9b1bdec9-cda2-4c03-8af1-a62744ae5bac
            correlation ID: ae3669ea-d6b2-47bf-90e6-616390cf31a2
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-hlfqz4 with version 4.21.39
Unexpected error:
    <*fmt.wrapError | 0x26fe4f09e7e0>: 
    failed to create HCP cluster np-version-upgrade-cluster-hlfqz4: failed waiting for cluster="np-version-upgrade-cluster-hlfqz4" in resourcegroup="rg-np-version-upgrade-hlfqz4-hxtt44szpfn8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9b1bdec9-cda2-4c03-8af1-a62744ae5bac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9b1bdec9-cda2-4c03-8af1-a62744ae5bac",
      "name": "9b1bdec9-cda2-4c03-8af1-a62744ae5bac",
      "status": "Failed",
      "startTime": "2026-10-22T03:09:57.381547287Z",
      "endTime": "2026-10-22T03:39:05.056394482Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: konnectivity-agent, redhat-operators-catalog, catalog-operator, cluster-version-operator, cluster-network-operator, dns-operator, csi-snapshot-controller-operator, redhat-marketplace-catalog, cluster-policy-controller, cluster-storage-operator, openshift-route-controller-manager, kube-controller-manager, ingress-operator, packageserver, olm-operator, kube-scheduler, hosted-cluster-config-operator, openshift-controller-manager, cluster-node-tuning-operator, cluster-image-registry-operator, community-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9b1bdec9-cda2-4c03-8af1-a62744ae5bac
            correlation ID: ae3669ea-d6b2-47bf-90e6-616390cf31a2
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server-proxy, router]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-4-22-gp6dgjrbtz82/cluster-candidate-4-22-qlh9hc should provision
Unexpected error:
    <*fmt.wrapError | 0x1de55d1398a0>: 
    failed to create HCP cluster cluster-candidate-4-22-qlh9hc: failed waiting for cluster="cluster-candidate-4-22-qlh9hc" in resourcegroup="rg-candidate-4-22-gp6dgjrbtz82" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cc15b2a-8360-41a2-8a2a-f0798be9df0d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cc15b2a-8360-41a2-8a2a-f0798be9df0d",
      "name": "8cc15b2a-8360-41a2-8a2a-f0798be9df0d",
      "status": "Failed",
      "startTime": "2026-10-22T02:21:54.045486495Z",
      "endTime": "2026-10-22T02:51:00.49750165Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cc15b2a-8360-41a2-8a2a-f0798be9df0d
            correlation ID: 1eeaec8e-068a-41c6-8cee-4c44e2977d7a
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-4-22-gp6dgjrbtz82/cluster-candidate-4-22-qlh9hc should provision
Unexpected error:
    <*fmt.wrapError | 0x1de55d1398a0>: 
    failed to create HCP cluster cluster-candidate-4-22-qlh9hc: failed waiting for cluster="cluster-candidate-4-22-qlh9hc" in resourcegroup="rg-candidate-4-22-gp6dgjrbtz82" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cc15b2a-8360-41a2-8a2a-f0798be9df0d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cc15b2a-8360-41a2-8a2a-f0798be9df0d",
      "name": "8cc15b2a-8360-41a2-8a2a-f0798be9df0d",
      "status": "Failed",
      "startTime": "2026-10-22T02:21:54.045486495Z",
      "endTime": "2026-10-22T02:51:00.49750165Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cc15b2a-8360-41a2-8a2a-f0798be9df0d
            correlation ID: 1eeaec8e-068a-41c6-8cee-4c44e2977d7a
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0x5228e164e00>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-d2fn5kshsl75" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e2b2e462-7fbf-427d-acfa-1649fda7fc18
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e2b2e462-7fbf-427d-acfa-1649fda7fc18",
      "name": "e2b2e462-7fbf-427d-acfa-1649fda7fc18",
      "status": "Failed",
      "startTime": "2026-10-22T03:11:10.501548627Z",
      "endTime": "2026-10-22T03:40:15.062794425Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-node-tuning-operator, konnectivity-agent, ingress-operator, openshift-route-controller-manager, cluster-network-operator, kube-controller-manager, cluster-policy-controller, cluster-storage-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, packageserver, cluster-version-operator, openshift-controller-manager, olm-operator, redhat-operators-catalog, community-operators-catalog, certified-operators-catalog, dns-operator, redhat-marketplace-catalog, kube-scheduler, hosted-cluster-config-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e2b2e462-7fbf-427d-acfa-1649fda7fc18
            correlation ID: ae820045-80be-4efc-9077-32c6901a632b
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0x5228e164e00>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-d2fn5kshsl75" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e2b2e462-7fbf-427d-acfa-1649fda7fc18
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e2b2e462-7fbf-427d-acfa-1649fda7fc18",
      "name": "e2b2e462-7fbf-427d-acfa-1649fda7fc18",
      "status": "Failed",
      "startTime": "2026-10-22T03:11:10.501548627Z",
      "endTime": "2026-10-22T03:40:15.062794425Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-node-tuning-operator, konnectivity-agent, ingress-operator, openshift-route-controller-manager, cluster-network-operator, kube-controller-manager, cluster-policy-controller, cluster-storage-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, packageserver, cluster-version-operator, openshift-controller-manager, olm-operator, redhat-operators-catalog, community-operators-catalog, certified-operators-catalog, dns-operator, redhat-marketplace-catalog, kube-scheduler, hosted-cluster-config-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e2b2e462-7fbf-427d-acfa-1649fda7fc18
            correlation ID: ae820045-80be-4efc-9077-32c6901a632b
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0x19243183a940>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-df27xfnfzvrk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92eda184-8a06-461e-9d73-222caa436db8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92eda184-8a06-461e-9d73-222caa436db8",
      "name": "92eda184-8a06-461e-9d73-222caa436db8",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:06.303940712Z",
      "endTime": "2026-10-22T02:03:06.782181855Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator, capi-provider, azure-cloud-controller-manager, csi-snapshot-controller-operator, catalog-operator, redhat-operators-catalog, certified-operators-catalog, cluster-image-registry-operator, control-plane-pki-operator, kube-controller-manager, machine-approver, cluster-network-operator, konnectivity-agent, redhat-marketplace-catalog, olm-operator, packageserver, cluster-policy-controller, kube-scheduler, cluster-version-operator, openshift-controller-manager, openshift-route-controller-manager, community-operators-catalog, hosted-cluster-config-operator, cluster-storage-operator, cluster-node-tuning-operator, dns-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, capi-provider deployment has 2 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92eda184-8a06-461e-9d73-222caa436db8
            correlation ID: 8174d018-ca0b-4942-ba1c-716f164824c9
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0x19243183a940>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-df27xfnfzvrk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92eda184-8a06-461e-9d73-222caa436db8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92eda184-8a06-461e-9d73-222caa436db8",
      "name": "92eda184-8a06-461e-9d73-222caa436db8",
      "status": "Failed",
      "startTime": "2026-10-22T01:34:06.303940712Z",
      "endTime": "2026-10-22T02:03:06.782181855Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator, capi-provider, azure-cloud-controller-manager, csi-snapshot-controller-operator, catalog-operator, redhat-operators-catalog, certified-operators-catalog, cluster-image-registry-operator, control-plane-pki-operator, kube-controller-manager, machine-approver, cluster-network-operator, konnectivity-agent, redhat-marketplace-catalog, olm-operator, packageserver, cluster-policy-controller, kube-scheduler, cluster-version-operator, openshift-controller-manager, openshift-route-controller-manager, community-operators-catalog, hosted-cluster-config-operator, cluster-storage-operator, cluster-node-tuning-operator, dns-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, capi-provider deployment has 2 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92eda184-8a06-461e-9d73-222caa436db8
            correlation ID: 8174d018-ca0b-4942-ba1c-716f164824c9
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-q2kkkp" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0x300bf5b7a840>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-q2kkkp: failed waiting for cluster="cp-ystream-upgrade-4-21-q2kkkp" in resourcegroup="rg-cp-ystream-upgrade-4-21-t4mn7sf29xrr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54a4385e-0b42-4987-acf3-1b0207b030a5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54a4385e-0b42-4987-acf3-1b0207b030a5",
      "name": "54a4385e-0b42-4987-acf3-1b0207b030a5",
      "status": "Failed",
      "startTime": "2026-10-22T02:23:48.751941498Z",
      "endTime": "2026-10-22T02:52:50.375994252Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54a4385e-0b42-4987-acf3-1b0207b030a5
            correlation ID: f3ca853a-64dc-4e52-ab57-050647054f81
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-q2kkkp" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0x300bf5b7a840>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-q2kkkp: failed waiting for cluster="cp-ystream-upgrade-4-21-q2kkkp" in resourcegroup="rg-cp-ystream-upgrade-4-21-t4mn7sf29xrr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54a4385e-0b42-4987-acf3-1b0207b030a5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54a4385e-0b42-4987-acf3-1b0207b030a5",
      "name": "54a4385e-0b42-4987-acf3-1b0207b030a5",
      "status": "Failed",
      "startTime": "2026-10-22T02:23:48.751941498Z",
      "endTime": "2026-10-22T02:52:50.375994252Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54a4385e-0b42-4987-acf3-1b0207b030a5
            correlation ID: f3ca853a-64dc-4e52-ab57-050647054f81
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0x262004875180>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-v4w4cbtgl6st" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0ffe01b2-d2c4-4e16-a6d6-042cbbeb51ec
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0ffe01b2-d2c4-4e16-a6d6-042cbbeb51ec",
      "name": "0ffe01b2-d2c4-4e16-a6d6-042cbbeb51ec",
      "status": "Failed",
      "startTime": "2026-10-22T02:27:27.45827201Z",
      "endTime": "2026-10-22T02:56:30.386782585Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0ffe01b2-d2c4-4e16-a6d6-042cbbeb51ec
            correlation ID: a0e42f8f-bf63-4670-b77c-e6ba06f19444
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0x262004875180>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-v4w4cbtgl6st" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0ffe01b2-d2c4-4e16-a6d6-042cbbeb51ec
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0ffe01b2-d2c4-4e16-a6d6-042cbbeb51ec",
      "name": "0ffe01b2-d2c4-4e16-a6d6-042cbbeb51ec",
      "status": "Failed",
      "startTime": "2026-10-22T02:27:27.45827201Z",
      "endTime": "2026-10-22T02:56:30.386782585Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0ffe01b2-d2c4-4e16-a6d6-042cbbeb51ec
            correlation ID: a0e42f8f-bf63-4670-b77c-e6ba06f19444
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
01:09:23 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-7pjtrh" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0x3fc646288640>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-7pjtrh: failed waiting for cluster="cp-ystream-upgrade-5-0-7pjtrh" in resourcegroup="rg-cp-ystream-upgrade-5-0-ph2qs5rvpjbv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1952a1f5-b390-468e-8207-de9b42085d84
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1952a1f5-b390-468e-8207-de9b42085d84",
      "name": "1952a1f5-b390-468e-8207-de9b42085d84",
      "status": "Failed",
      "startTime": "2026-10-22T02:23:50.650035503Z",
      "endTime": "2026-10-22T02:52:50.371348301Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1952a1f5-b390-468e-8207-de9b42085d84
            correlation ID: 13cd48b6-3df3-4a5e-9127-3bc8b9f3564d
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-7pjtrh" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0x3fc646288640>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-7pjtrh: failed waiting for cluster="cp-ystream-upgrade-5-0-7pjtrh" in resourcegroup="rg-cp-ystream-upgrade-5-0-ph2qs5rvpjbv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1952a1f5-b390-468e-8207-de9b42085d84
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1952a1f5-b390-468e-8207-de9b42085d84",
      "name": "1952a1f5-b390-468e-8207-de9b42085d84",
      "status": "Failed",
      "startTime": "2026-10-22T02:23:50.650035503Z",
      "endTime": "2026-10-22T02:52:50.371348301Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1952a1f5-b390-468e-8207-de9b42085d84
            correlation ID: 13cd48b6-3df3-4a5e-9127-3bc8b9f3564d
            client request ID: unavailable
occurred